axrea.

Lending · the full example · worked examples

The headline is good news. The credit team looks underneath.

An illustrative engagement, not a client's: a specialist property lender's credit team measures one decision in its credit papers, with its AI assistant and without. Measure, design, show what it is worth, re-test.

The lender
A specialist property lender, and its credit team.
The decision
The valuation review in a credit paper, ending in "refer" or "accept".
The AI
A document assistant that reads the valuation report and pre-fills the review.
The people
Four credit analysts. Two senior credit officers set the answers and score.
The loans
30 past loans, then 20 more held in reserve.

What the lender believes

The assistant saves time, and it is accurate.

Both beliefs have evidence. Analysts are quicker with the assistant. And the lender's own quarterly check has it right on 291 of 300 fields.

How the review is written

  1. The AIReads the valuation report
  2. The AIPre-fills the review and the recommendation
  3. The analystReads the review and changes any entry
  4. The analystAccepts

The report is a click away. Nothing has to be opened before the analyst accepts.

The check measures the assistant. Nobody has measured the decision the analyst signs.

Measure the decision · the first look

With the assistant, and without.

The lender tests thirty past loans, chosen to include awkward reports. Analysts write the review alone and with the assistant. Its senior credit officers score every one against answers fixed before the assistant is run.

Reviews with a material error

No worse with the assistant.

Alone35%
With the assistant28%

21 of 60 alone, 17 of 60 with the assistant. A test of this size cannot tell those two apart. On the headline, the assistant is doing no harm.

Median analyst minutes a review

And far quicker.

Alone44.5
With the assistant17

Minutes released are capacity. What counts is what the analyst does with them.

It would be easy to stop here. This team asks what the average is made of.

What measuring shows

Four things the average does not show.

The same thirty loans record what the analysts do, not only what they get right.

Reliance

When the assistant is wrong, the analyst usually goes with it.

15of 28 readings

On the 14 loans where the pre-filled review carries a material error, 15 of the 28 analyst readings carry one too.

The unread source

The report is opened less than a third of the time.

16of 60 readings

The assistant's summary holds five key points at most. A special assumption or a caveat outside them stays in the report.

The default

"Accept" is left standing where the policy says refer.

5of 30 readings where policy says refer

Each time, a special assumption or a caveat is missing from the same review. The recommendation follows the entries above it.

Agreement

Analysts agree with each other more. They are not right more.

Alone90%
With the assistant97%

Two analysts reach the same recommendation on 29 of 30 loans with the assistant, against 27 of 30 alone. Both readings begin from the same pre-filled answer.

The quarterly check is not wrong. It counts fields the assistant extracts. This counts reviews an analyst accepts.

Design the way of working

Six designs on the table. Three chosen.

Six designs from current research are set beside what the first look finds, each with its evidence, its cost and how it could backfire. The lender's senior credit officers choose, and its head of credit agrees.

Before

  1. The AIReads the report
  2. The AIPre-fills the review
  3. The analystChecks the review
  4. The analystAccepts

After

  1. The AIMarks each required item found or not found
  2. The AIPuts lease terms in their own fields, with the sums done outside the AI
  3. The analystLists what the report says, then confirms each entry
  4. The analystAccepts

Change 1

Found, and not found

The assistant marks what it found, and what it did not, against a list the senior credit officers own.

Change 2

Lease terms kept apart

Term to break and term to expiry become separate fields. The calculation moves to the lender's own spreadsheet.

Change 3

Reasons on the record

Before accepting, the analyst lists what they are relying on: the report's special assumptions and caveats, from the report itself.

The design they turn down

The expert commits first.

The analyst records a view from the report before the assistant's review is shown. It is the first lever on our own list. The lender says no.

The design not chosen

  1. The analystReads the key terms and records a view
  2. The AIIts review is revealed
  3. The analystReconciles the two
  4. The analystAccepts

Their reason

It gives back the time the assistant saves.

What is argued

That the analyst's own list is a checklist, and a checklist can be ticked unread. The head of credit hears both views and keeps the list.

What the second look shows

The list is confirmed with nothing on it in 4 of 40 readings.

We set out the price of each design. The people who run the credit team choose.

The new design, measured · the second look

Twenty new loans. Both ways of working.

Loans the analysts have not seen, each read under the way of working as it is and under the new design, and scored the same way.

Missed referrals, as it is

"Accept" where the policy says refer.

6of 20 readings

Missed referrals, the new design

None.

0of 20 readings

The errors the three changes are aimed at
In the review as acceptedAs it isNew design
Special assumption not recorded2 of 60 of 6
Caveat not carried into the review3 of 120 of 12
Term to the earliest break wrong3 of 120 of 12
"Accept" where the policy says refer6 of 200 of 20

Counts are readings where the error could arise. Two things no change reaches: a wrong rent figure pre-filled by the assistant, and two tenants in one group not added together. Both are written up for the lender, with where they arise.

Show what it is worth

Slower, in the places that matter.

The new design costs analyst time. The lender can now see how much, and what it is missing without it.

Analyst minutes a review

17.5to29

Median at the second look, with the checking counted.

Analyst hours a year

123to203

On the lender's own volume, about 420 reviews a year. Hours are capacity, and the lender chooses where to spend them.

A missed referral is a loan accepted on a value or an income nobody questioned. The lending example shows what one can be worth: a loan to value of 60% on one reading, and 70.6% on the other.

The recommendation is to change the way of working. The lender's decision is now a plain one: are the minutes worth paying?

They are a price the lender can see, to accept or decline before the new design goes into daily use.

Re-test

Tools move. The same loans, run again.

The assistant is updated during the first test. The lender is not told. The same past loans, run again, show when the way of working has stopped being right.

Run it again, on more loans

Twenty loans show a direction, not a size. The new design stays in test until the next run.

Keep the loans and the answers

Fifty loans, the agreed answers and the scoring rules stay with the lender, ready to run again.

Ask the vendor for notice

The lender asks to be told before the assistant changes, and runs the loans again when it does.

Your own team

Get the pack for your team.

Ten pages. Pages 2 to 4 take five minutes.

Back to the lending example

Get the pack for your team

Thirty minutes with our founder, not a salesperson. Bring one kind of decision, with no client details, and we will sketch how it would be measured.

Arrange the 30 minutes