Recognize Biased Evidence

Welcome to Recognize Biased Evidence! Today you will shift focus from the numbers themselves to the evidence behind them. A claim can show every number openly and frame them fairly and still mislead, because the underlying data may have been gathered in a slanted way. No amount of careful arithmetic can rescue evidence that quietly leaves out failures or compares groups that were never alike to begin with. In this lesson, you will learn to:

• Recognize survivorship bias by asking which failures, dropouts, or removed cases were filtered out before the data was collected, since a sample made of only survivors can look far rosier than reality.
• Uncover the invisible group behind a confident-sounding conclusion, because once you account for the cases that never made it into view, an apparent pattern often weakens or disappears entirely.
• Evaluate unfair comparisons by checking whether two groups differ in some important way besides the factor being studied, because a hidden difference can explain the result all on its own.

Imagine you walk through a historic neighborhood and admire buildings that are over a hundred years old. Their thick walls and handcrafted details might tempt you to conclude that "they just built things better back then." But pause: you are only looking at the buildings that lasted. The cheaply constructed structures from the same era already crumbled and were demolished decades ago. The sample is filtered, and the failures are invisible.

This is the core trap you will study first. When the evidence you see has already been filtered by success, survival, or visibility, the missing failures can completely change the picture.

Survivorship bias occurs when you draw conclusions from a group that has passed through some selection filter while ignoring all the cases that did not make it through. The result is an overly optimistic or distorted view, because the evidence you examine is not representative of the full picture.

Here is a concrete example. A freelancing platform surveys its active users and proudly reports: "Our freelancers earn an average of $90,000 per year." That sounds impressive. But "active users" means people who are still freelancing on the platform. Anyone who tried freelancing, struggled to find clients, and quit is no longer counted.

Suppose 100 people started freelancing on the platform three years ago. Today, 30 are still active and earn an average of $90,000. The other 70 gave up; while they were active, they averaged $25,000. The platform's headline reflects only the survivors:

The claim is not necessarily false. The problem is that the platform has shown only the survivors. By excluding those who left, it creates a much more optimistic picture than a complete accounting would provide.

The key to recognizing survivorship bias is asking one question: "What cases are missing, and could they change the conclusion?" Usually the missing group consists of failures, dropouts, or cases that were removed by some process before the data was collected. Here are a few common patterns:

In each case, the conclusion sounds reasonable until you remember the group that never made it into the sample. Once you account for those missing cases, the apparent pattern often weakens or disappears entirely.

The second form of biased evidence involves comparing two groups that differ in some important way besides the factor being studied. When the groups are not truly comparable, any observed difference might be caused by the hidden factor rather than the one in the spotlight.

Consider a magazine that compares two SAT prep courses:

• Course A students improved their scores by an average of 120 points.
• Course B students improved their scores by an average of 60 points.

At first glance, Course A looks twice as effective. But what if Course A's students started much lower, averaging 400 points below the top score, while Course B's students started only 150 points below it? Students in Course A began much farther below the maximum possible score, giving them more room to improve. Students in Course B were already much closer to the top of the scale.

Because the groups differed before the courses even began, you can't conclude that the courses themselves caused the difference. The starting scores provide an alternative explanation. The original comparison was unfair because the two groups started from very different positions. The hidden factor — starting score — explained the gap rather than the course itself.

Before accepting a comparison, ask were the groups similar before the thing being studied occurred?

Unfair comparisons appear whenever groups are selected, or self-selected, in ways that create built-in differences. A classic example is comparing hospital death rates. Hospital B might report a higher mortality rate than Hospital A, but if Hospital B is a specialized trauma center treating the most severe cases, its patients arrive in worse condition. The severity of incoming cases, not the quality of care, could explain the gap.

When you encounter a comparison between two groups, a short set of questions helps test fairness:

1. How were the groups formed? Were participants randomly assigned, or did they choose (or get chosen) based on characteristics that could affect the outcome?
2. Do the groups differ in any important starting condition? Look for differences in age, severity, income, experience, or other relevant factors.
3. Could that hidden difference, rather than the stated factor, explain the result? If yes, the comparison does not reliably support its conclusion.

If any of these questions raises a concern, you should look for adjusted data or simply note that the comparison is inconclusive on its own. As you practiced in the previous lesson, asking "What's missing?" remains one of your strongest tools. Here, what is missing is not a number but a fair playing field.

In this lesson, you explored two ways that evidence itself can be biased even when all the numbers are visible and correctly calculated. Survivorship bias filters out failures, leaving you with a misleadingly rosy sample, while unfair comparisons pit groups against each other without accounting for important differences that could explain the result. In both cases, the remedy is the same: ask what is missing and whether the groups being examined truly represent the full picture.

Up next, you will put these ideas into action across a series of practice exercises. You will identify survivorship bias in everyday conclusions, uncover the invisible groups whose absence distorts a claim, evaluate whether compared groups are truly fair, and explain the hidden factors that can flip a comparison on its head. Let's see how well you can separate solid evidence from the misleading kind!