Triage and Observability
Triage and Observability 🧭
Triage and observability help you investigate data-quality incidents without treating a signal as a confirmed cause.
In this lesson, you will learn to:
- Distinguish point-in-time quality checks from continuous observability signals of freshness, volume, schema, distribution, lineage, and operational context
- Apply a severity rubric across impact, scope, urgency, detectability, recurrence, and uncertainty to prioritize an incident
- Communicate a triage decision that separates symptom, evidence, hypothesis, and confirmed cause
Your checks passed at 6am. At 9am the regional revenue figure is visibly wrong and three people are in your inbox. Was the check wrong, or was it faithfully answering a question nobody had asked about this failure? In the previous lesson, you wrote expectations and placed controls. This lesson is about the messier half: what you do when reality disagrees with them, how you decide which disagreement gets worked first, and how you describe it without quietly inventing a cause.
Point-in-Time Checks and Continuous Signals 📡
| Approach | What it assesses | What it can reveal | Boundary |
|---|---|---|---|
| Point-in-time quality check | A defined rule against data at a specified moment | Whether a supplier code conforms to the reference list, for example | It detects violations of the conditions it was designed to test. It does not surface unrelated failure modes by itself. |
| Continuous observability signals | Whether the flow is behaving outside its normal pattern over time | Emerging deviation that warrants investigation | They narrow the field. They do not confirm a cause. |
| Observability signal | What it reveals |
|---|---|
| Freshness | Whether data is current and arrived when expected |
| Volume | An unexpected change in record quantity |
| Schema | A structural change in fields, types, or required elements |
| Distribution | A shift in the pattern of values, even when individual values remain valid |
| Lineage | The upstream dependencies on which a figure relies |
| Operational context | Relevant events, such as a release or regional systems upgrade |
Observability signals narrow the field; they never close it. Lineage showing a recently changed feed alongside an overnight upgrade is a coincidence in time with a plausible story attached. Treating that as an answer is a common failure in triage notes, especially for people who are right often enough to stop checking.
Ranking What Matters ⚖️
Most organizations rank incidents by who noticed them. An executive sees a 34% drop on a dashboard and it becomes priority one, while four thousand customers reading disagreeing loyalty balances stay at priority three because no dashboard reflects them. The triage factors make that ordering arguable rather than political.
Use the six factors to make a ranking defensible. Rate each qualitatively, then use the pattern of evidence, not a mechanical total, to assign a severity tier, owning domain, and escalation path.
| Factor | Question to ask | How it affects priority |
|---|---|---|
| Impact | How severe is the consequence? | Greater harm increases priority. |
| Scope | How far does the issue reach? | More affected people, decisions, or reports increase priority. |
| Urgency | How quickly will the consequence land? | Less time to prevent harm increases priority. |
| Detectability | Would consumers notice unaided? | Low detectability is worse because an incorrect figure can travel unchallenged. |
| Recurrence | Is this a pattern rather than a one-off event? | Repeated failure increases priority and points to an unresolved weakness. |
| Uncertainty | How much is still unknown? | Score it openly. Uncertainty can justify escalation, but it is not proof of a cause. |
A severity tier is a locally defined routing decision, not a score that the six factors calculate. Define the tiers for the service before using them. A context may define Tier 1 as active customer or regulatory harm, or broad and immediate decision risk; Tier 2 as material but contained harm or a broad decision risk that can be bounded while investigated; and Tier 3 as a contained, low-urgency defect with a workaround. The exact definitions can differ, but they must be stated where the triage decision is made.
Notice what the factors do to two apparently competing incidents. A dashboard drop can have high visibility and high uncertainty with a plausible benign explanation. A loyalty discrepancy can have confirmed customer-facing consequence, real scope, and low uncertainty. Ranking the second below the first is defensible only if you can say why.
Saying It Without Guessing 🧾
A triage communication has four layers, and collapsing them destroys architectural trust:
- Symptom: what the consumer observed
- Evidence: what checks and signals actually recorded
- Hypothesis: a candidate explanation, explicitly labeled as such
- Confirmed cause: an explanation verified by evidence
In this exchange, Nina is a reporting stakeholder seeking a quick explanation, while Dan is the data-operations reviewer responsible for recording only what the evidence supports.
> **Nina:** So the checkout-conversion drop was yesterday's payment-gateway change. > > **Dan:** That's a hypothesis, not a cause. What we have is the symptom, conversion down 18%, and the evidence: freshness normal, completed-order volume down 15%, no schema change, lineage pointing at a changed promotions feed, and a gateway change in the same window. > > **Nina:** Which is the gateway change. > > **Dan:** It's a change that overlaps in time. If I write "cause: gateway" in the note, that phrasing reaches the executive update by lunchtime, and nobody reopens it if the promotions feed is the real problem. > > **Nina:** Fine. What would confirm it? > > **Dan:** Order counts for the promotions that changed. Until then, it stays labeled a hypothesis.Dan does not refuse to name a suspect. He refuses to promote it a level.
The takeaway is this: hold a ranking you can defend separately from a cause you have not yet earned. Two quick pattern checks come next to sharpen the signals and factors, and then you will write the triage assessment itself: three concurrent incidents, six factors, and one ranking you have to argue rather than assert. The question to carry in is whether you can demote the incident the executive noticed and still be believed.
