Recognize Biased Evidence
Introduction 🎉
Welcome to Recognize Biased Evidence! Today you will shift focus from the numbers themselves to the evidence behind them. A claim can show every number openly and frame them fairly and still mislead, because the underlying data may have been gathered in a slanted way. No amount of careful arithmetic can rescue evidence that quietly leaves out failures or compares groups that were never alike to begin with. In this lesson, you will learn to:
- Recognize survivorship bias by asking which failures, dropouts, or removed cases were filtered out before the data was collected, since a sample made of only survivors can look far rosier than reality.
- Uncover the invisible group behind a confident-sounding conclusion, because once you account for the cases that never made it into view, an apparent pattern often weakens or disappears entirely.
- Evaluate unfair comparisons by checking whether two groups differ in some important way besides the factor being studied, because a hidden difference can explain the result all on its own.
Imagine you walk through a historic neighborhood and admire buildings that are over a hundred years old. Their thick walls and handcrafted details might tempt you to conclude that "they just built things better back then." But pause: you are only looking at the buildings that lasted. The cheaply constructed structures from the same era already crumbled and were demolished decades ago. The sample is filtered, and the failures are invisible.
This is the core trap you will study first. When the evidence you see has already been filtered by success, survival, or visibility, the missing failures can completely change the picture.
Survivorship Bias 🪖
Survivorship bias occurs when you draw conclusions from a group that has passed through some selection filter while ignoring all the cases that did not make it through. The result is an overly optimistic or distorted view, because the evidence you examine is not representative of the full picture.
Here is a concrete example. A freelancing platform surveys its active users and proudly reports: "Our freelancers earn an average of $90,000 per year." That sounds impressive. But "active users" means people who are still freelancing on the platform. Anyone who tried freelancing, struggled to find clients, and quit is no longer counted.
Suppose 100 people started freelancing on the platform three years ago. Today, 30 are still active and earn an average of $90,000. The other 70 gave up; while they were active, they averaged $25,000. The platform's headline reflects only the survivors:
The claim is not necessarily false. The problem is that the platform has shown only the survivors. By excluding those who left, it creates a much more optimistic picture than a complete accounting would provide.


