Choosing Evaluation Metrics

You run a prompt, the output looks correct, and the evaluation reports a score of zero. Nothing is broken—the model answered "The capital of France is Paris" while the metric demanded the single word "Paris". Moments like this are where most early evaluation work goes wrong: the prompt gets blamed for a decision that was really made when the metric was chosen.

Here you will learn what the available metrics actually measure, how to match one to the quality bar your job requires, and how to read a score and an output together so you can tell a metric mismatch from a genuine prompt failure. The habit you walk away with is simple: decide what a successful answer is allowed to look like before you decide how to score it.

To measure how well a prompt performs, you need more than just an output; you need a target. An expected answer (sometimes called a "ground truth" or "reference") represents the ideal response for a given input. By comparing the model's actual output against this reference using a mathematical rule—a metric—you can turn qualitative behavior into a quantitative score. This transformation allows you to move from subjective impressions like "this feels better" to a data-driven conclusion based on objective performance.

That only works if you know what each rule is actually comparing.

Now that you have seen why repeated runs matter, you need a fair way to decide whether each output meets the job. Prompt Eval pairs each case with an expected answer and applies a metric, which is the rule used to compare the model's output with that reference. For example, an input asking for the capital of France might have "Paris" as its expected answer. Whether "The capital of France is Paris" passes depends on the metric.

Exact Match requires the output to match the expected answer exactly. It is useful when the prompt should return only the city name, with no extra words or formatting.

Exact Match CI means case-insensitive exact match. It treats "Paris" and "paris" as equivalent, but it can still reject "The capital is Paris" when the expected answer is only "Paris".

Contains passes when the expected answer appears somewhere in the output. It allows an answer such as "The capital of France is Paris", but it does not judge whether the surrounding text is helpful or contradictory.

String similarity is a softer comparison. Instead of treating every non-exact answer as an immediate failure, it estimates how close the output is to the expected text and may provide partial credit. This can help you see near matches or changes in wording, but textual closeness is not the same as factual correctness. For example, "Pari" resembles "Paris" even though it is not the correct city.

Word-overlap F1 compares the words shared by the output and the expected answer. It balances two questions: how much of the expected wording appeared, and how much of the output was relevant to that expected wording. An output such as "Paris, France" shares the key word with "Paris", but extra words can lower its score. This makes Word-overlap F1 useful when wording and length may vary, while still allowing partial credit for incomplete or wrong answers.

The first three metrics usually behave like pass or fail checks. String similarity and Word-overlap F1 are softer, so they can produce a range of scores. That range describes textual overlap or closeness. It does not prove that the model gave the right answer.

Note: This is not an exhaustive list of all evaluation methods. While many other metrics exist—including model-based evaluators (LLM-as-a-judge) and specialized linguistic scores—we will focus on these core metrics for this lesson and the upcoming practices.

Knowing what each metric measures is only half the decision; the other half is knowing what your job demands. Choose the quality bar before choosing the metric. Ask what a successful answer is allowed to look like. If the job requires the city name only, Exact Match is the clearest choice. If capitalization should not matter, Exact Match CI relaxes that one requirement without allowing extra wording.

If a complete sentence is acceptable, Contains is often a better fit. For the expected answer "Paris", both "Paris" and "The capital of France is Paris" can satisfy the basic text requirement. However, Contains can also accept a sentence such as "Paris is not the capital of France" because the expected city name appears inside it. The metric checks for presence, not meaning.

String similarity or Word-overlap F1 may be useful when partial wording overlap is genuinely informative. They are less suitable as a strict pass rule for a factual job, because a wrong answer can look similar to the expected answer or share some of its words. If the required fact is a stable name inside otherwise flexible wording, Contains may be safer and easier to interpret than a soft similarity score.

A common mistake is reading a low score as a failure of the prompt when it is really a mismatch of the metric. If your quality bar allows conversational responses but you use Exact Match, a correct answer like "The capital is Paris" scores zero: the prompt succeeded, and the rule was too restrictive to capture it.

Mismatches run in both directions, and the second one is easier to miss:

• Too strict produces false failures. A zero sits next to an output you would have accepted by hand.
• Too loose produces false passes. A one sits next to an output you would have rejected—Contains accepting "Paris is not the capital of France", or String similarity rewarding "Pari".

False passes are the more dangerous case, because a zero invites investigation and a one rarely does. Spot-check a few passing outputs, not only the failing ones.

To tell the two situations apart, look at the output: if the required fact is there and correctly stated, the rule is the problem; if the fact is missing, wrong, or contradicted, the prompt is the problem and loosening the metric would only hide it. When every case fails the same mechanical way—all lowercase, all wrapped in a sentence—suspect the metric before the reasoning.

The useful move is to treat a surprising score as a reason to inspect the quality bar and the output together, not as automatic evidence that the prompt failed. Doing that inspection well is easiest on a small set of cases.

Once you have expected answers and a metric, run the prompt on a small set of factual cases. Prompt Eval will show the model's output and the score for each case. Read those two pieces of evidence together. A score without the output cannot tell you why a case passed or failed.

To see how a metric affects your results, you can change the metric and run the evaluation again. However, remember what you learned about variation: because you are triggering a new run, the model might provide slightly different wording this time.

To truly "isolate" the metric, you must compare the new score against the output that generated it. If the model says "The capital is Paris" in both runs, but the score jumps from 0 to 1 when you switch from Exact Match to Contains, you have isolated the metric's effect. If the output itself changed, the new score reflects both the metric change and the model's natural variation.

Also check the expected answer itself. If the reference is wrong, incomplete, or ambiguous, the case is not strong evidence about the prompt. A useful diagnosis asks three questions:

• Did the expected answer represent the quality bar? A reference that is wrong, vague, or narrower than the job will mislead every score built on it.
• Did the metric measure that bar? A strict rule applied to a flexible job, or a soft rule applied to a factual job, produces scores that do not mean what you think.
• Did the model output meet it? Only after the first two questions check out is a low score genuine evidence about the prompt.

The core shift is this: a score is not a verdict on your prompt: it is the result of three choices working together—the expected answer, the metric, and the output. Change any one of them and the number changes, even when the model's behavior did not.

So set the quality bar first, then pick the rule that matches it. Exact Match and Exact Match CI for strict formats, Contains when flexible wording around a stable fact is acceptable, and the softer metrics only when partial overlap is genuinely informative. When a score surprises you, read the output next to it and ask the three diagnostic questions before touching the prompt.

Next, you'll put this into practice: matching metrics to jobs, running a tiny multi-case evaluation in Prompt Eval, and explaining whether a surprising result comes from the metric or from a genuine prompt miss.