If you are worried about “redlining” you’d better understand statistical significance
The perennial question facing all banks is, “What volume of lending in Majority-Minority census tracts and mortgages extended to minority home borrowers must we do to avoid redlining allegations by bank regulators?” The answer provided by regulators is found in their application of the concept of “Statistical Significance.” Therefore, it behooves bankers to understand statistical significance, how it is applied and its strengths and weaknesses and when it can lead to misleading conclusions.
Bankers cannot ignore statistically significant results. In this article we refer to the application of statistical significance analysis to potential redlining cases based on geographic considerations, but it also may be applied to performance lending to minority applicants.
So, what is statistical significance and how is it applied by banking regulators?
Statistical significance is a tool to identify statistical results that are so extreme as to not likely be the byproduct of pure chance. It is important to understand that the statistical significance model is based only on the raw chance or probability of a potential outcome without consideration of factors that can affect the lender’s results, or the peer data used to determine the benchmark against which a bank is being compared. It is truly a “numbers game”. This is a strength and a weakness at the same time, and it is imperative to understand the implications.
The technique uses “Two Factor Analysis” applied to the number of applications or originations processed by an institution to determine the likelihood of the institution’s penetration rate in the market’s Majority-Minority census tracts (“MMCTS”) is statistically significant when compared to the market’s (or peers) MMCT penetration rate. Examiners use a 5% “level of significance” below which a lender’s result would be deemed statistically significant. This means that there is a probability of 5% or less that the lender’s performance is not by chance (“bad luck” when below the market) and therefore would be deemed statistically significant (meaningful). A formula is applied that uses the market average penetration rate and that adjusts that rate based on the number of applications processed by the examined institution to determine if the result is statistically significant.
For example, if an institution processes a total of 100 applications in a market area and 10 of those applications are in MMCTs but the market has 10,000 applications of which 1,500 are in MMCTs is the institution’s MMCT penetration rate “statistically significant”?
Based on the numbers in the example above, the institution’s penetration rate of 10% is not statistically significant even though it is below the market’s MMCT penetration rate of 15%. Using the above numbers, the lender would need a MMCT penetration rate of only a little more than 9% or more to avoid a statistically significant result.
But suppose the lender processed 1,000 applications and 100 were in the MMCTs. Both the market (15%) and the institution (10%) have the same MMCT penetration rates as in the first example. One would assume therefore that the lender’s MMCT penetration rate would still not be statistically significant. But that is not the case. In the second situation the lender would need to process more than 131 (or 13.1%) of its applications in the MMCTs to avoid a statistically significant result. It’s still below the market average MMCT penetration rate, but the minimum to avoid a statistically significant result is not as far below the market average as in the first example (the thresholds to avoid a SS result is a little more than 13% in the second example and only a little more than 9% in the first example).
Why would a lender need to attain only a 9% penetration rate in the MMCTs compared to the market’s 15% MMCT penetration rate to avoid a statistically significant result based on 100 applications, but when the lender processes 1,000 applications it would need to process more than 13% (131) applications in the MMCTs to avoid a statistically significant result?
Why is a MMCT penetration rate of only 9% not statistically significant in one case but a 13% rate would be statistically significant in the second case even though the mean market MMCT penetration rate is 15% in both cases. The answer is that with smaller numbers there is a greater chance of error. But with a larger number of applications the closer the MMCT penetration would need to be to the market. If you think about it, it makes sense. The closer a lender’s volume is to the market volume the closer its performance should be to the market. But with smaller volumes allowance must be made for more uncertainty.
Again, it cannot be overstated that the analysis is based on pure chance of a combination of the institution’s potential MMCT penetration rates without consideration of any other factors that may affect the results of the institution’s, or the market’s, MMCT penetration rate. In the real world, this doesn’t happen. But the point of statistical significance is to determine the probability of an institution’s performance based only on chance.
Again, the idea is this, if an institution’s statistical result is so extremely low compared to the market average MMCT penetration rate then it is highly unlikely to be due to chance and therefore an indication that the situation bears further investigation. Typically, regulators infer that this means they can say the institution may be doing something they shouldn’t be doing or may not be doing something they should be doing, otherwise their results would not compare so unfavorably to the competition. The problem is that this response reflects regulatory prejudice that overlooks potential bias in the market or peer data that constitute the benchmarks against which the lender is being compared. The fault may not be the lender’s doing. It may be due to a bias in the market or peer data.
So, what are the potential biases in the market or peer data that may distort the market results and cause a misleading conclusion?
First, it is imperative that the “market” to which the analysis is applied must be a true market for the institution. Prior to the “Combatting Redlining Initiative” as announced and practiced beginning in October 2021, regulators had applied the concept of a “REMA” or Reasonably Expected Market Area. Regulators explained the factors they consider when determining the REMA for an institution. Those factors included a bank’s marketing campaigns, its CRA assessment area, the geographic dispersion of its mortgage applications etc. But after the Combatting Redlining Initiative the regulators announced a radically different approach with REMA’s in most cases, being no smaller than an entire MSA or MD. For many community banks with limited resources, that is a market far too large to be considered a level playing field. A REMA that extends beyond the practical service area of an institution invalidates a major assumption underlying the application of statistical significance.
Second, extreme outliers in the peer data can skew the market average MMCT penetration rates. Regulators define peers as lenders that process anywhere from 50% to 200% of a lender’s application activity. But in many cases mortgage companies constitute a large part of the peer population as defined by regulators. Many of those lenders are subprime lenders with MMCT penetration rates far above those of bank lenders. In one situation we observed a peer group of 45 lenders of which 22 were mortgage companies. Not only did the bank being examined have a statistically significant result, so too did 22 of the lenders in the group, including 16 of the 23 banks. This means that nearly half the lenders in the peer group had statistically significant low results. But one would expect only a small (5%) percentage of the lenders to fail achieve a high enough volume to avoid a statistically significant result due to chance. This suggests an underlying bias in the peer group. When the mortgage company peers were removed 12 banks (plus the bank being examined) failed to achieve enough volume in the MMCTs to avoid a SS result. Within the group of banks, however, there was an extreme outlier that achieved an MMCT penetration rate almost 3 times the average for that group. When that institution was removed from the peers only 4 banks failed to achieve enough volume in the MMCTs to avoid a low SS result. That is a more normal result approximating what would be expected if there is no bias in the peer group.
The lesson for banks with statistically significant results is to examine the REMA used for the analysis and to validate the legitimacy of the peer group by testing all peer lenders for statistical significance. If the number of lenders falling below the statistical significance threshold is large, the validity of the peer group is questionable.
