Interpreting Prompt Comparisons

You already know what a fair prompt comparison requires: the same cases, the same expected answers, the same metric, the same run count, and the same model, with only the prompt itself allowed to change. It's tempting to think that once that setup is fair, the results interpret themselves — run both prompts, glance at whichever number is bigger, and call it done. But two prompts can post the exact same overall score while behaving completely differently underneath, and a prompt with a better overall score can still be quietly worse on a case that matters. Reading an A/B result well is a separate skill from designing a fair one.

You'll walk through actually enabling Prompt B in the Prompt Evaluation Simulator, running it alongside Prompt A under one shared setup, and reading what comes back — first the aggregate outcome, then the case-by-case pattern hiding inside it. The habit you'll build: compare overall results, then inspect case-level improvements and regressions, then inspect the actual outputs to understand what changed — never stop at the headline number.

As a quick reminder from the previous lesson: fairness means the case set, expected answers, metric, run count, and model are all frozen, and the prompt is the only thing allowed to differ. Assume that discipline is already in place; what's new here is actually switching the simulator into compare mode.

Enabling Prompt B adds a second prompt field next to Prompt A. This isn't a second experiment bolted onto the first one. Once compare mode is on, the case set, expected answers, metric, model used and run count you've already configured apply identically to both prompts — one shared setup, evaluated twice, not two separate setups that happen to sit side by side.

Picture a store-hours normalization job. Each case gives the model a messy input like Mon-Fri 9-5, Sat 10-2, closed Sun and expects a cleaned-up, structured output such as Mon-Fri 09:00-17:00, Sat 10:00-14:00, Sun Closed. Prompt A gives a short instruction: "Normalize the store hours." Prompt B is more explicit: "Normalize the store hours using 24-hour HH:MM time. Return only the normalized hours." Both prompts are evaluated on the same 5 store-hours cases, the same expected answers, Exact Match (case-insensitive) as the metric, and 3 runs each, on the same model.

You can picture the shared setup feeding both prompt slots like this:

        ┌───────────────────────────────┐
        │     Shared setup (frozen)     │
        │  cases · expected answers ·   │
        │     metric · runs · model     │
        └───────────────┬───────────────┘
                ┌───────┴───────┐
                ▼               ▼
            Prompt A         Prompt B
      ("Normalize the      ("Normalize the store
       store hours.")       hours using 24-hour
                            HH:MM time. Return only
                            the normalized hours.")

Only the boxes on the bottom — the prompt wording — change between A and B. Everything above them stays identical for both.

With the setup shared and only the wording different, the next question is simple: what does the simulator actually hand back once both prompts have run?

Once you run both prompts, the simulator produces an overall mean score for Prompt A and an overall mean score for Prompt B. At the aggregate level, there are exactly three possible outcomes:

• A higher — Prompt A's overall mean is greater than Prompt B's.
• B higher — Prompt B's overall mean is greater than Prompt A's.
• Tie — the two overall means are equal.

Suppose Prompt A's mean comes out to 0.73 and Prompt B's comes out to 0.80. That's a straightforward "B higher" outcome. But ties can be less obvious than they sound. Say Prompt A scores 1, 1, 0, 1 across its four cases — mean = 3/4 = 0.75. Prompt B scores 1, 0, 1, 1 across the same four cases — mean = 3/4 = 0.75 as well. The overall means tie exactly, even though the two score lists aren't identical case-by-case at all.

That tie is the seam worth noticing: A and B just landed on the same number by getting different cases right. A single mean can be the same story told two different ways, or two different stories wearing the same number — and the only way to tell which one you're looking at is to look inside it.

Opening Case details for both prompts lets you line up their scores case by case instead of only comparing the one summary number. Three patterns describe what you'll find at the case level:

• Improvement — Prompt B scores higher than Prompt A on that specific case.
• Regression — Prompt B scores lower than Prompt A on that specific case.
• No change — Prompt A and Prompt B score the same on that case, whether both pass, both fail, or both land at the same partial score.

Go back to the store-hours job with 5 cases. Suppose Prompt A scores 1, 0, 1, 0, 1 (mean = 3/5 = 0.6) and Prompt B scores 1, 1, 0, 0, 1 (mean = 3/5 = 0.6) — an overall tie. Laid out case by case, the picture looks like this:

The overall tie was hiding one improvement and one regression that happened to cancel each other out. That raises a sharper question: what happens when the overall mean doesn't tie, but one prompt is still quietly losing ground on a case somewhere underneath its higher score?

Two things are true about aggregate results that go beyond a simple higher/lower/tie read:

• A prompt can post a higher overall mean while still regressing on one or more individual cases — being better on average doesn't mean being better everywhere.
• Equal overall means don't guarantee identical behavior — as the tie above showed, two prompts can match on the surface while succeeding and failing on completely different cases.

Suppose Prompt A scores 1, 1, 0, 1, 0 across five cases → mean = 3/5 = 0.6. Prompt B scores 1, 0, 1, 1, 1 → mean = 4/5 = 0.8. At the aggregate level, B wins clearly. But look case by case: Case 2 is a regression (A scored 1, B scored 0), while Cases 3 and 5 are improvements (0 to 1 on each). Prompt B's higher mean is real — but part of it was bought by trading away a case Prompt A used to get right. That trade-off is completely invisible if you only compare 0.6 to 0.8.

Spotting a regression this way tells you that something changed on a given case. To understand why, you have to stop reading scores and start reading the actual text the model produced.

For any case flagged as an improvement, a regression, or even a no-change, open Case details and read the actual outputs from both prompts side by side. Score differences tell you where to look; the outputs tell you what happened — a formatting slip, a dropped instruction, or a different-but-still-wrong answer.

Take the Case 2 regression from the previous section. Prompt A's output might read Mon-Fri 09:00-17:00, Sat 10:00-14:00, Sun Closed — matching the expected answer exactly. Prompt B's output on that same case might read Mon-Fri 9-5, Sat 10-2, Sun Closed — logically correct hours, but failing Exact Match (case-insensitive) because the model didn't apply the 24-hour HH:MM format the prompt asked for. Reading the text shows exactly what the score alone couldn't: Prompt B's added instruction helped on some cases but got dropped or ignored on this one.

Previous unit asked "is this comparison fair?". With that fairness already in place, the next question is: "we ran a fair comparison — what do the results actually tell us?" The answer is never fully contained in a single mean. A higher overall score for Prompt B can still mask a real regression on a case Prompt A used to nail, and a perfect tie can mask two prompts succeeding on entirely different cases.

Compress it into the habit to carry forward: compare overall results, then inspect case-level improvements, regressions, and unchanged cases, then inspect the actual outputs to understand what changed — and when runs are repeated, treat small gaps with caution rather than certainty. Aggregate first, case-level second, outputs last, every time.

The practices ahead put this directly into your hands: a short preview of how compare mode works and what stays shared between A and B, a hands-on run of Prompt A against Prompt B on the store-hours job under one frozen setup, and a chance to read Results and Case details to classify what actually happened — whether one prompt clearly won, the two tied, or a mix of case-level improvements and regressions was sitting underneath the headline number all along.