Monitoring Drift and Unintended Optimization

A rising headline score can hide worsening outcomes, so a weekly review needs evidence that challenges the score rather than celebrates it.

In this lesson, you will learn to:

• Identify the weekly indicators that reveal drift, harm, and unintended optimization.
• Analyze a rising productivity score alongside worsening resolution times and complaints.
• Facilitate a review that ends with a named, evidence-based pause trigger.

Here is a question worth sitting with before you read any further. If a live system's headline score improved for five weeks running, would you read that as evidence it is working, or as evidence that you need to go looking somewhere else?

Most leaders read it as good news. But think about what that score actually is: it is the one number the system was designed to move. Of course it went up. The interesting question is what happened to everything the score was supposed to stand for. In the last lesson you saw how a system's decisions quietly reshape the records it learns from next. Monitoring is the discipline of catching that while it is still small.

You do not need a monitoring department. You need four kinds of evidence reviewed, in order, every week by someone allowed to say something unwelcome:

1. Case mix: Are the requests, customers, or documents arriving this month broadly like the ones the system studied? When they are not, you have drift: the world moved and the system did not. Nothing is broken, and yet the answers are quietly getting worse.

2. Customer outcomes: Review the thing the score is a stand-in for: how long people wait, whether their problem got solved, and whether they came back.

3. Harm signals: Customer complaints, escalations, and refunds tell you one story. Complaints from your own frontline tell you another, and leaders discard those far too readily as change resistance.

4. Behavior: Look for signs that people are working to the score rather than to the customer. That is unintended optimization, and it is not cheating. It is what any sensible person does when they are measured on one thing and judged by it. If closing a case early makes an agent's number look healthy, some of them will close cases early, and the number will improve while the service degrades.

Notice what these four have in common. Three of them cannot be found in the score at all.

Use this sequence to test whether the score still represents the outcome you care about.