Fair Prompt Comparisons

Imagine you've written two versions of a prompt for classifying support tickets by priority. You run Prompt A once, get a score of 0.6. You run Prompt B once, get a score of 0.8. It's tempting to declare Prompt B the winner and move on. But what if Prompt B was tested on easier tickets? What if it was graded with a looser metric? What if it got five tries and Prompt A only got one? Any of those differences could produce a score gap all by itself — with the prompt wording having nothing to do with it. A comparison that looks like evidence can quietly be no evidence at all.

You'll learn to spot that trap before it fools you: exactly which conditions have to stay identical when comparing two prompts, what to call it when one of those conditions is allowed to drift (a confound), and how to explain — in plain language — why a comparison with a confound can't be trusted as evidence. The habit you'll leave with: before asking "which prompt won?", ask "was this test even fair?"

You already know that running the same prompt multiple times under fixed conditions — same input, same metric, same run settings — is how you find out whether that prompt behaves consistently. Comparing two different prompts follows the identical discipline, extended by exactly one step. The prompt is now allowed to change, because that's the entire point of an A/B comparison. But everything else — every other condition that was frozen when you were just repeating one prompt — has to stay just as frozen now that two prompts are involved.

Here's the core rule: in a fair comparison, the prompt is the only deliberate variable.

Picture a ticket-priority job: given a support ticket's text, the model must label it High, Medium, or Low. Prompt A and Prompt B are two different sets of instructions for doing that labeling. If you evaluate both prompts under the same conditions — the identical set of ticket cases, the identical expected answers, the same metric, and the same number of runs — the prompt is the only deliberate difference between the two setups. Any score difference can then be investigated as evidence about the effect of the prompt, while repeated runs help you account for normal model variation.

The next question is: which conditions, exactly, need to be frozen for that logic to hold?

In any evaluation, underlying runtime and generation settings must stay fixed so nothing shifts in the background. In our simulator, those runtime settings are already locked for you, leaving five conditions you configure that must be identical across Prompt A's evaluation and Prompt B's evaluation. If even one of them differs, you lose the ability to credit a score gap to the prompt.

• Test-case set — the exact same ticket inputs, in the same number, used for both prompts. Not a similar set. The same one.
• Expected answers — the same reference labels used to score both prompts' outputs, so neither prompt is being held to a different standard of "correct."
• Metric — the same scoring method applied to both, for example Exact Match (case-insensitive) for both prompts, never Exact Match CI for one and Contains for the other.
• Run count — the same number of runs per case for both prompts, for example 3 runs each, not 1 run for one and 5 for the other.
• Model — the same model generating outputs for both prompts. Switching models isn't something you'll do hands-on in this unit, but the same freezing logic applies to it, and a later course returns to it directly.

Here's what a fair setup looks like in practice: 5 ticket cases, Exact Match CI as the metric, 3 runs, and the same model, used identically for both Prompt A and Prompt B. Say Prompt A's mean lands at 0.73 and Prompt B's mean lands at 0.80. Because all conditions matched, the comparison gives you meaningful evidence about whether the wording difference affected performance. You would still look across repeated runs before deciding that a small gap is reliable.

Note on Stochasticity and Uncontrollable Noise: LLMs are inherently stochastic, meaning their behavior can shift due to pure randomness. In production, even subtle external variables like the time of day a prompt is run can slightly alter performance due to provider-side load balancing, dynamic hardware allocation, or API routing. Because absolutely identical conditions in every possible dimension are impossible to guarantee, you don't need to chase perfect environmental parity. Scale should help: create a lot of test cases and ensure significant amount of repeated runs is performed, and those small inconsistencies will naturally average out.

That checklist tells you what should stay fixed; the next question is what happens when one of these five items is allowed to drift instead — that's where a comparison stops being evidence.

When one of the five checklist conditions isn't frozen, it introduces a confound: an alternative explanation for the score difference, sitting alongside the prompt, that you can no longer rule out. Once a confound is present, you can't say the prompt caused the gap — because something else changed too.

Here are five common ways those conditions can be confounded, with the arithmetic worked out so you can see the mechanism rather than just a label:

• Case-set confound — Prompt A is scored on cases with results 1, 1, 0, 1, 0 (mean 3/5 = 0.6). Prompt B is scored on a different, easier set of cases with results 1, 1, 1, 1, 0 (mean 4/5 = 0.8). The 0.2 gap might reflect the fact that B's cases were simply easier — not that B's wording is better.
• Expected-answers confound — Both prompts produce the output High on the same ticket. When Prompt A is scored, that ticket's reference label is Medium, so the output counts as 0; when Prompt B is scored, the same ticket's reference label is recorded as High, so the identical output counts as 1. Prompt B ends up ahead only because it was graded against a different answer key — the gap reflects which labels were called "correct," not the prompt wording.
• Metric confound — Suppose both prompts produce the output "The priority is High" for an expected answer of High. Under Exact Match CI, that output scores 0 because of the extra words. Under Contains, it scores 1 because High appears as a substring. If Prompt A is graded with Exact Match CI and Prompt B is graded with Contains, Prompt B looks better for no reason other than being measured with a looser ruler.
• Run-count confound — Prompt A is tested with a single run, producing one score of either 0 or 1. Prompt B is tested with 5 runs and averaged, producing something like a mean of 0.6. A one-run score is far more sensitive to a single lucky or unlucky output than a five-run mean is, so the two numbers aren't measuring the same amount of evidence — comparing them directly is comparing apples to a much steadier average.
• Model confound — If Prompt A runs on one model and Prompt B runs on a different one, any score gap could reflect a difference in model capability rather than a difference in prompt wording. This is the same freezing principle as the other three, just applied to the model instead of the cases, metric, or run count.

Of these five, one tends to slip past people unnoticed, because it doesn't feel like cheating — it feels generous.

Suppose you give Prompt A 3 test cases and Prompt B those same 3 cases plus 2 additional ones. Nothing about the extra cases is illegitimate — they're real tickets with correctly labeled expected answers. But the comparison is still broken, because Prompt A was never given a chance to attempt those extra 2 cases. Any lift they produce in Prompt B's mean can't be separated from "which cases happened to be included," which means it can't be credited to the prompt wording.

Here's the arithmetic:

• Prompt A, 3 cases, scores 1, 0, 1 → mean = 2/3 ≈ 0.667
• Prompt B, 5 cases (the same 3, plus 2 new, easier ones), scores 1, 0, 1, 1, 1 → mean = 4/5 = 0.8

At a glance, Prompt B looks stronger by about 0.133. But look only at the 3 cases both prompts actually share: both score 1, 0, 1, giving both an identical mean of 0.667. The entire apparent advantage came from two cases Prompt A never even had the opportunity to attempt — the extra cases produced the lift, not the prompt.

The shift here is small but consequential: instead of asking "did this prompt score well?", ask "was this comparison even fair?" first. A higher score only means something once you've confirmed nothing besides the prompt was allowed to change.

Compress it into one habit: freeze the case set, the expected answers, the metric, the run count, and the model — the prompt is the only thing allowed to move. If a score gap shows up alongside any other difference, name the confound before you believe the winner.

The next unit puts this discipline to work by actually running Prompt A against Prompt B in the simulator. Before that, the practices ahead will have you name what needs to stay fixed for the ticket-priority job, match flawed setups to the confounds they create, and write out, in your own words, why two prompts can both report the same number of cases and still fail the fair-compare checklist when the case sets are not identical.