Welcome to Recognize Biased Evidence! Today you will shift focus from the numbers themselves to the evidence behind them. A claim can show every number openly and frame them fairly and still mislead, because the underlying data may have been gathered in a slanted way. No amount of careful arithmetic can rescue evidence that quietly leaves out failures or compares groups that were never alike to begin with. In this lesson, you will learn to:

• Recognize survivorship bias by asking which failures, dropouts, or removed cases were filtered out before the data was collected, since a sample made of only survivors can look far rosier than reality.
• Uncover the invisible group behind a confident-sounding conclusion, because once you account for the cases that never made it into view, an apparent pattern often weakens or disappears entirely.
• Evaluate unfair comparisons by checking whether two groups differ in some important way besides the factor being studied, because a hidden difference can explain the result all on its own.

Imagine you walk through a historic neighborhood and admire buildings that are over a hundred years old. Their thick walls and handcrafted details might tempt you to conclude that "they just built things better back then." But pause: you are only looking at the buildings that lasted. The cheaply constructed structures from the same era already crumbled and were demolished decades ago. The sample is filtered, and the failures are invisible.

This is the core trap you will study first. When the evidence you see has already been filtered by success, survival, or visibility, the missing failures can completely change the picture.

The key to recognizing survivorship bias is asking one question: "What cases are missing, and could they change the conclusion?" Usually the missing group consists of failures, dropouts, or cases that were removed by some process before the data was collected. Here are a few common patterns:

In each case, the conclusion sounds reasonable until you remember the group that never made it into the sample. Once you account for those missing cases, the apparent pattern often weakens or disappears entirely.

The second form of biased evidence involves comparing two groups that differ in some important way besides the factor being studied. When the groups are not truly comparable, any observed difference might be caused by the hidden factor rather than the one in the spotlight.

Consider a magazine that compares two SAT prep courses:

• Course A students improved their scores by an average of 120 points.
• Course B students improved their scores by an average of 60 points.

At first glance, Course A looks twice as effective. But what if Course A's students started much lower, averaging 400 points below the top score, while Course B's students started only 150 points below it? Students in Course A began much farther below the maximum possible score, giving them more room to improve. Students in Course B were already much closer to the top of the scale.

Because the groups differed before the courses even began, you can't conclude that the courses themselves caused the difference. The starting scores provide an alternative explanation. The original comparison was unfair because the two groups started from very different positions. The hidden factor — starting score — explained the gap rather than the course itself.

Before accepting a comparison, ask were the groups similar before the thing being studied occurred?

Unfair comparisons appear whenever groups are selected, or self-selected, in ways that create built-in differences. A classic example is comparing hospital death rates. Hospital B might report a higher mortality rate than Hospital A, but if Hospital B is a specialized trauma center treating the most severe cases, its patients arrive in worse condition. The severity of incoming cases, not the quality of care, could explain the gap.

When you encounter a comparison between two groups, a short set of questions helps test fairness:

1. How were the groups formed? Were participants randomly assigned, or did they choose (or get chosen) based on characteristics that could affect the outcome?
2. Do the groups differ in any important starting condition? Look for differences in age, severity, income, experience, or other relevant factors.
3. Could that hidden difference, rather than the stated factor, explain the result? If yes, the comparison does not reliably support its conclusion.

If any of these questions raises a concern, you should look for adjusted data or simply note that the comparison is inconclusive on its own. As you practiced in the previous lesson, asking "What's missing?" remains one of your strongest tools. Here, what is missing is not a number but a fair playing field.

In this lesson, you explored two ways that evidence itself can be biased even when all the numbers are visible and correctly calculated. Survivorship bias filters out failures, leaving you with a misleadingly rosy sample, while unfair comparisons pit groups against each other without accounting for important differences that could explain the result. In both cases, the remedy is the same: ask what is missing and whether the groups being examined truly represent the full picture.

Up next, you will put these ideas into action across a series of practice exercises. You will identify survivorship bias in everyday conclusions, uncover the invisible groups whose absence distorts a claim, evaluate whether compared groups are truly fair, and explain the hidden factors that can flip a comparison on its head. Let's see how well you can separate solid evidence from the misleading kind!