Reading Confidence Scores

You've screened an input list and hunted for hindsight. Now the tool is running, and it hands you a number next to every case: 91, 64, 38. That number can feel like certainty, and it is one of the easiest things in this course to misread.

In this lesson, you will learn to:

• Explain what a confidence score does and does not promise.
• Choose whether a case should Act, Verify, or Escalate by weighing confidence, stakes, and reversibility.
• Translate those choices into practical instructions a frontline team can use.

A confidence score is the system's own read on how strong the signal is for this one case. Nothing more. It is closer to "this case looks a lot like the ones I've seen before" than to "I am right about this one."

That distinction carries three practical consequences:

1. A score should not automatically be treated as a probability of correctness. What it means depends on how the system defines and calibrates it — checks that a score of 90 really behaves like a 90. A case scored 91 is not guaranteed correct nine times out of ten in your live queue, especially if your live cases differ from the ones the system studied.
2. A score says nothing about stakes. The system does not know whether the decision costs three pounds or forty thousand, and it does not know whether you can undo it tomorrow. It scores the pattern, not the consequence.
3. On unfamiliar cases, even a high score may be less trustworthy than it appears. On a case unlike anything the system has seen, it can still return a high number because it is matching the closest thing it recognises rather than admitting unfamiliarity.

Here is how that overreading sounds in a conversation between Meredith, an invoice-approval lead, and Dan, a colleague reviewing the same case:

• Meredith: It came back at 88. That's higher than my best approver's accuracy, so I'd just push it through.
• Dan: Your best approver would look at the amount before deciding. Does the system know this one is thirty thousand pounds and can't be clawed back once it's paid?
• Meredith: ...no. It doesn't see the amount at all, does it.
• Dan: It sees the pattern. You see the pattern and the consequence. That's why the 88 is an input to your decision, not the decision.

Notice that Dan never argued the score was wrong. He separated the strength of the signal from the size of the consequence, which is the whole move.