Question Evidence from Small Samples
Introduction 🎉
Welcome to Question Evidence from Small Samples! Over the previous four lessons you learned how independent events behave, saw why the gambler's fallacy and hot-hand belief mislead us, used regression to the mean to interpret extreme performances, and discovered why randomness naturally produces clusters, coincidences, and false patterns. All of those ideas share a common thread: random processes create surprising-looking outcomes far more often than we expect. In this final lesson, you'll turn to one of the most practical applications of that insight. By the end of this lesson, you will have learned to:
- Explain why small samples produce extreme and unstable results, and recognize that randomness has not had enough observations to settle down when the data is limited.
- Convert percentages back into raw counts to expose thin evidence, and always ask "out of how many?" before trusting a number.
- Apply practical evaluation questions about sample size, actual counts, and replication to decide whether a trend or difference is genuine or just noise from limited data.
These skills are essential for clear thinking in a world full of data. By mastering these concepts, you will learn to pause before trusting a number and recognize when an apparent signal is really just noise from too little evidence.
The Problem With Small Amounts of Evidence 🌱
Imagine two restaurants on the same street. Restaurant A has a 4.9-star rating based on 7 reviews. Restaurant B has a 4.4-star rating based on 1,200 reviews. Which one is actually better?
Most of us are drawn to the higher number, but that 4.9 rating is standing on very thin ground. With only 7 opinions, a single unhappy diner could drop the average dramatically. Restaurant B's rating, built on over a thousand experiences, is far more stable. This everyday example captures the core lesson: the less data behind a number, the less that number tells us.
Randomness produces streaks, clusters, and extreme results all the time, as we saw throughout this course. Small samples are where those effects hit hardest, because there simply is not enough data for the extremes to balance out.
Let's make this concrete with a simple thought experiment. Suppose a fair coin is flipped repeatedly and we record the percentage of heads.
- In 10 flips, getting 80% heads (8 out of 10) is unusual but not shocking.
- In 100 flips, getting 80% heads (80 out of 100) is virtually impossible with a fair coin.
- In 1,000 flips, the percentage of heads will almost certainly land between 47% and 53% — the result barely strays from 50% because the large sample leaves randomness very little room to push it around.
The pattern is clear: the smaller the sample, the wilder the results. This is not a flaw in the data collection; it is a basic property of randomness. Small samples simply have not run long enough for that settling to occur.
Notice why this happens. In a small sample, a single observation carries enormous weight — flipping just one or two coins the other way swings the percentage wildly. In a large sample, each observation is only a tiny slice of the whole, so no single outcome can move the result very far. That is the heart of why small samples are so unstable: there is simply too little data for the extremes to cancel out.
