Lending · the full example · worked examples
The headline is good news. The credit team looks underneath.
An illustrative engagement, not a client's: a specialist property lender's credit team measures one decision in its credit papers, with its AI assistant and without. Measure, design, show what it is worth, re-test.
- The lender
- A specialist property lender, and its credit team.
- The decision
- The valuation review in a credit paper, ending in "refer" or "accept".
- The AI
- A document assistant that reads the valuation report and pre-fills the review.
- The people
- Four credit analysts. Two senior credit officers set the answers and score.
- The loans
- 30 past loans, then 20 more held in reserve.
What the lender believes
The assistant saves time, and it is accurate.
Both beliefs have evidence. Analysts are quicker with the assistant. And the lender's own quarterly check has it right on 291 of 300 fields.
How the review is written
- The AIReads the valuation report
- The AIPre-fills the review and the recommendation
- The analystReads the review and changes any entry
- The analystAccepts
The report is a click away. Nothing has to be opened before the analyst accepts.
The check measures the assistant. Nobody has measured the decision the analyst signs.
Measure the decision · the first look
With the assistant, and without.
The lender tests thirty past loans, chosen to include awkward reports. Analysts write the review alone and with the assistant. Its senior credit officers score every one against answers fixed before the assistant is run.
Reviews with a material error
No worse with the assistant.
21 of 60 alone, 17 of 60 with the assistant. A test of this size cannot tell those two apart. On the headline, the assistant is doing no harm.
Median analyst minutes a review
And far quicker.
Minutes released are capacity. What counts is what the analyst does with them.
It would be easy to stop here. This team asks what the average is made of.
What measuring shows
Four things the average does not show.
The same thirty loans record what the analysts do, not only what they get right.
Reliance
When the assistant is wrong, the analyst usually goes with it.
15of 28 readings
On the 14 loans where the pre-filled review carries a material error, 15 of the 28 analyst readings carry one too.
The unread source
The report is opened less than a third of the time.
16of 60 readings
The assistant's summary holds five key points at most. A special assumption or a caveat outside them stays in the report.
The default
"Accept" is left standing where the policy says refer.
5of 30 readings where policy says refer
Each time, a special assumption or a caveat is missing from the same review. The recommendation follows the entries above it.
Agreement
Analysts agree with each other more. They are not right more.
Two analysts reach the same recommendation on 29 of 30 loans with the assistant, against 27 of 30 alone. Both readings begin from the same pre-filled answer.
The quarterly check is not wrong. It counts fields the assistant extracts. This counts reviews an analyst accepts.
Design the way of working
Six designs on the table. Three chosen.
Six designs from current research are set beside what the first look finds, each with its evidence, its cost and how it could backfire. The lender's senior credit officers choose, and its head of credit agrees.
Before
- The AIReads the report
- The AIPre-fills the review
- The analystChecks the review
- The analystAccepts
After
- The AIMarks each required item found or not found
- The AIPuts lease terms in their own fields, with the sums done outside the AI
- The analystLists what the report says, then confirms each entry
- The analystAccepts
Change 1
Found, and not found
The assistant marks what it found, and what it did not, against a list the senior credit officers own.
Change 2
Lease terms kept apart
Term to break and term to expiry become separate fields. The calculation moves to the lender's own spreadsheet.
Change 3
Reasons on the record
Before accepting, the analyst lists what they are relying on: the report's special assumptions and caveats, from the report itself.
The design they turn down
The expert commits first.
The analyst records a view from the report before the assistant's review is shown. It is the first lever on our own list. The lender says no.
The design not chosen
- The analystReads the key terms and records a view
- The AIIts review is revealed
- The analystReconciles the two
- The analystAccepts
Their reason
It gives back the time the assistant saves.
What is argued
That the analyst's own list is a checklist, and a checklist can be ticked unread. The head of credit hears both views and keeps the list.
What the second look shows
The list is confirmed with nothing on it in 4 of 40 readings.
We set out the price of each design. The people who run the credit team choose.
The new design, measured · the second look
Twenty new loans. Both ways of working.
Loans the analysts have not seen, each read under the way of working as it is and under the new design, and scored the same way.
Missed referrals, as it is
"Accept" where the policy says refer.
6of 20 readings
Missed referrals, the new design
None.
0of 20 readings
| In the review as accepted | As it is | New design |
|---|---|---|
| Special assumption not recorded | 2 of 6 | 0 of 6 |
| Caveat not carried into the review | 3 of 12 | 0 of 12 |
| Term to the earliest break wrong | 3 of 12 | 0 of 12 |
| "Accept" where the policy says refer | 6 of 20 | 0 of 20 |
Counts are readings where the error could arise. Two things no change reaches: a wrong rent figure pre-filled by the assistant, and two tenants in one group not added together. Both are written up for the lender, with where they arise.
Show what it is worth
Slower, in the places that matter.
The new design costs analyst time. The lender can now see how much, and what it is missing without it.
Analyst minutes a review
17.5to29
Median at the second look, with the checking counted.
Analyst hours a year
123to203
On the lender's own volume, about 420 reviews a year. Hours are capacity, and the lender chooses where to spend them.
A missed referral is a loan accepted on a value or an income nobody questioned. The lending example shows what one can be worth: a loan to value of 60% on one reading, and 70.6% on the other.
The recommendation is to change the way of working. The lender's decision is now a plain one: are the minutes worth paying?
They are a price the lender can see, to accept or decline before the new design goes into daily use.
Re-test
Tools move. The same loans, run again.
The assistant is updated during the first test. The lender is not told. The same past loans, run again, show when the way of working has stopped being right.
Run it again, on more loans
Twenty loans show a direction, not a size. The new design stays in test until the next run.
Keep the loans and the answers
Fifty loans, the agreed answers and the scoring rules stay with the lender, ready to run again.
Ask the vendor for notice
The lender asks to be told before the assistant changes, and runs the loans again when it does.
Your own team
Get the pack for your team.
Ten pages. Pages 2 to 4 take five minutes.
Thirty minutes with our founder, not a salesperson. Bring one kind of decision, with no client details, and we will sketch how it would be measured.
Arrange the 30 minutes