Trusted, easy, transparent… and problematic
Scorecards have traditionally helped financial institutions assess a loan applicant’s creditworthiness and help guide the decision to lend. To create one, experts familiar with an institution’s business help select and weight a series of variables to evaluate an individual’s credit risk. For example, a microfinance institution offering small business loans might include socio-demographic profile, payment history, business activity, collateral, and other financial activity such as savings and insurance.
The result is a trusted ‘expert scorecard’ that requires very little technical expertise to implement. The simplest can be calculated by hand, helping the institution quickly determine if the applicant meets the necessary threshold for a loan. And due to this simplicity, expert scorecards can be easy to understand for customers and regulators, and even – if the institution desires – rendered totally transparent.
As simple and straightforward as they are, however, expert scorecards are increasingly seen as problematic. They suffer from the bias of the humans that create them, may expose financial institutions to more risk, and possibly even compromise their mission of financial inclusion. Let’s see why.
Bias in expert scorecard construction
The challenges of expert scorecards start with their construction. Whether it’s choosing the sample pool to create the algorithm, or the variables selected and the weight they’re given, there are a number of ways that human biases can compromise their effectiveness as a tool for evaluation.
Variable selection bias
For a credit scorecard to be successful, the final credit score must objectively correlate with future repayment behaviours. However, the variables selected by individual experts based on their experience may not be the ones with the most predictive power for the entire customer base. Their inclusion in the scorecard may result in a system biased heavily towards the subjective experience of one or a few experts. In our experience, it’s not unusual to come across selected scorecard variables that don’t correlate – or even, worse, that inversely correlate – with the likelihood of future default.
Weighting bias
It’s not just the variables selected that can be problematic, but also how they’re weighted. For example, when a credit institution chooses to place extra weight on socio-demographic variables with strong collinearity (i.e. correlation), the scorecard can produce aberrant results. We’ve sometimes observed this situation: Customer A, with an excellent repayment and savings history is scored as not eligible; Customer B, with a similar profile, less collateral coverage and several arrears on their credit is scored as eligible – simply due to the weight of certain socio-demographic variables for which Customer B performed better. These problems with variable selection and weighting are counterproductive by ultimately increasing the risk for the credit institution.
Sample selection bias
Another important source of error that is often overlooked is sample selection bias. This occurs when the default data on which the algorithm is based comes from, for example, customers who already have a credit history and who have therefore already been selected beforehand by a credit committee.
This can produce a vicious circle: the algorithm continues to favour the same type of customers already selected by the credit committee, while it continues to exclude customers similar to those not selected, even though they haven’t had the chance to prove themselves yet. Several studies have also shown excluding higher risk customers from the data sample results in less accurate predictions for all applicants.
Inflexibility leads to unfairness
In addition to sample bias in construction, the inflexible construction of expert scorecards and the credit scoring system can result in other kinds of unfairness for customers.
By their very nature, scorecards are inflexible. They’re a static representation of a person’s financial situation at a given moment. While that situation can change – even rapidly – the person’s score does not. And people are rarely given the opportunity or incentive to positively influence the variables that determine their score. For instance, a positive change of repayment behaviour on the customer’s part will not necessarily be rewarded by a better score.
Additionally, credit thresholds determined by expert scorecards may fail to take into consideration the size of the request, with applicants classified as risky or not regardless of how much they want to borrow. However, an applicant’s ability to repay is never absolute, but relative to the amount requested: a customer judged a high risk for a standard small business loan may still be perfectly able to repay a nano loan.
These inflexibilities and oversights can make the scorecard result something of a sentence – once ineligible, always ineligible. They can unfairly reinforce exclusion, and lead to frustration on the part of the customer and ultimately affect their relationship with the bank.
Unnecessarily complex and unscalable
Finally, despite the potential for simplicity described above, the scorecard’s actual simplicity is conditioned by the number and complexity of its variables. It’s not uncommon to find scorecard models built with dozens of variables with multiple thresholds, resulting in extreme complexity for the scorer.
For example, if a scorecard has 12 variables, each with a different threshold (e.g. for age: 1 point if the customer is less than 20 years old, 5 points if they’re 20-30 years old, 10 points if they’re 30+, and so on) it can result in thousands of possible configurations. Each one of those configurations leads to a unique decision – and the sheer number of decisions necessary increases the likelihood of error.
To make matters worse, variables are not always clearly defined and sometimes require additional data collection, which results in wasted time for loan officers.
This means that expert scorecards can be a challenge to successfully use, scale up and automate, making it difficult to handle large numbers of applicants in a timely manner.
The end of the expert scorecard?
Despite the perceived trust, ease and transparency of expert scorecards, this approach has many limitations which can cost the financial institution in wasted time, missed opportunities, and poor risk management. Customer inclusion and satisfaction can also be severely compromised because of construction bias and a customer’s inability to improve their score when their financial situation changes, or through good behaviour.
Thankfully, however, we’re now in a much better position to overcome the limitations of expert scorecards. Recent developments in behavioural and data science have seen the emergence of a range of data-intensive and AI solutions that can help financial institutions improve their credit scoring – and help them say goodbye to the limitations of the expert scorecard.