Choosing Better System Signals

A decision system follows the signal it is rewarded to move, so the signal choice determines what the system treats as success.

In this lesson, you will learn to:

• Distinguish the business outcome you want from the easy activity measure a system will chase.
• Analyze how a click-focused recommendation rule can weaken long-term customer value.
• Design value signals, warning indicators, and written limits before build work begins.

This lesson starts at the earliest and cheapest point of influence: the moment someone decides what the system is being rewarded for.

Here's the question worth sitting with: when your data team says "we're optimizing for engagement," what exactly have you agreed to?

Every decision system needs something to aim at. That aim gets written down as a measurable signal, a number the system can count. And here is the uncomfortable part: whatever signal you choose does not merely describe success, it becomes success. The system will pursue it with a literalness no human employee would. A person told to increase clicks understands the unstated conditions: don't mislead people, don't wreck trust, don't burn the relationship. A system understands none of that. It optimizes the number you gave it, and only that number.

So ask yourself: is the signal on the table the outcome you want, or a convenient stand-in for it? Clicks, calls handled, tickets closed, minutes watched — these are activity measures. They are attractive because they arrive fast, count cleanly, and fit on a slide. The outcome you actually want — the customer's need genuinely resolved, the customer still with you in a year — is slower, messier, and harder to report. That asymmetry is exactly why the wrong signal usually wins by default rather than by argument.

Use this map to keep a convenient activity measure from becoming the only definition of success.

Natalie and Milo are reviewing the product brief together. Natalie is protecting the reported target, while Milo tests whether the target also protects customers.

• Natalie: The brief says the objective is total clicks. That's what we're building toward.
• Milo: And if the fastest route to more clicks is putting alarming headlines at the top of every feed?
• Natalie: Then clicks go up. Which is the target, so... good?
• Milo: Good on the slide. What would we see if it was going badly for customers at the same time?
• Natalie: Honestly? Nothing on our dashboard. We don't measure that.
• Milo: Then the dashboard can only ever tell us we succeeded.

Notice what Milo does not do. He doesn't argue the target is wrong on principle. He asks what evidence of harm would even be visible, and the silence answers the question.