Understanding LLM Judges

Unit 2 ended on a problem it couldn't solve: {"priority":"low"} is perfectly valid JSON. It parses, it has the right key, it passes a Valid JSON check — and if the ticket described a production outage, it's still the wrong answer. The check confirmed the shape and said nothing about the content underneath.

This lesson introduces the evaluator built for that gap: LLM-as-a-Judge, which compares an output against its Expected Answer and scores whether they mean the same thing. You'll see what the judge compares, how to run it, and how to read its scores without over-trusting them.

Every metric you've used so far compares surface text or structural form. That leaves two failure modes.

A passing check can hide a wrong answer. Valid JSON asked whether {"priority":"low"} parses, and answered correctly. It was never asked whether "low" was right. Regex Match has the same limit: matching a shape says nothing about the value inside it.

A failing score can hide a right answer. String similarity and word-overlap F1 do look at content, but only as shared characters or shared words. Consider a paraphrased-FAQ job with four cases:

Mean: 0.05. Three of four answers were correct, and the only case earning points was the one that contradicts its Expected Answer — it just happened to reuse the word "refunds." The metric detected shared vocabulary, and here vocabulary and meaning came apart completely.

Both failure modes measure something other than meaning. That's the gap.

LLM Judge is an evaluator that uses a language model to assess output quality.

Under the hood, there is no magic: the platform passes the generated output and the Expected Answer into an evaluation prompt that instructs the judge model how to score them. Conceptually, that prompt looks something like this:

"You are an evaluation judge. Compare the Output against the Expected Answer. Score from 0.0 to 1.0 based on whether they agree in factual meaning, regardless of wording differences."

Because it is driven by a prompt:

• It needs an Expected Answer for every case. The judge prompt explicitly compares the two texts; it doesn't know what is correct on its own.
• It's pre-configured here. Teams in practice can write custom judge prompts to evaluate tone, safety, or conciseness. In this course, the judge is set up specifically for relevance and semantic agreement.
• You select it like any other metric, directly from the metric control.

Which to reach for:

You might wonder why you wouldn't just ask an LLM judge to check JSON validity too. Modern LLMs can often spot simple syntax errors, but doing so has two clear downsides:

1. Reliability: A model is an approximator; it can make mistakes or overlook subtle syntax errors that a programmatic JSON parser catches with 100% certainty.
2. Cost and efficiency: Running an LLM costs money and takes noticeable time. Using an expensive, slower model call to check syntax that a free, instant programmatic check handles perfectly is an inefficient spend.

And the rule that spans all of them: a metric only confirms what it checks. Regex Match confirms a shape, Valid JSON confirms parseability, and a passing judge score tells you nothing about formatting or structure. Different questions, different answers.

Pick LLM Judge from the metric control, fill in Expected Answer for every case, and run. Results land where you already expect them: overall mean, score distribution, per-case results, and Case details showing each output beside its score.

Scores use the same 0.00 to 1.00 scale as every other metric, and they're graduated rather than pass/fail — the judge can land in the middle when an output is partly right.

Run the four-case FAQ job from above with LLM Judge instead of F1, and the picture inverts: the three correct paraphrases land in the top band, the refund contradiction in the bottom. The exact mean depends on how the judge scores each case, but it will reflect semantic agreement far better than 0.05 did.

Because the judge is itself an LLM running a prompt, treat its score as evidence, not an infallible verdict. It can occasionally be swayed by confident phrasing or miss a subtle contradiction. A 0.9 means "the judge evaluated this as semantically consistent" — strong evidence, but not a guarantee. When a score surprises you in either direction, open Case details and read the output against the Expected Answer yourself.

Middle-band scores are worth reading, not averaging away. A 0.4–0.7 usually means one of two things: the output is genuinely partial, or your Expected Answer is vague enough that "correct" is ambiguous. Only you can tell which — and if the ambiguity is yours, fix the Expected Answer.

Scores can move between runs, and it matters what moved. Two separate sources:

• Evaluator variability — the same output scored 0.85 one run and 0.90 the next, because the judge itself is an LLM.
• Output variability — the model under test produced a different answer, so the score followed.

Case details tells them apart: compare the output text across runs. Identical output with a different score means the judge wiggled; different output means the model did. Small wiggles inside a band are normal. A case that swings across bands — 0.2 one run, 0.9 the next — is a flag to read by hand, because either the outputs are genuinely unstable or the Expected Answer is too vague to score consistently.

Case details also shows you why a score happened. Take Expected Answer "You can update your email in Account Settings" against output "Go to your account settings page to change your email." The judge scores it high because the instruction matches. A Contains check would also have passed it — but only because the string "account settings" appeared, which would have passed a contradicting answer just as happily. Same verdict, different justification, and only Case details reveals which one you got.

The shift is from checks that confirm form to an evaluator that estimates meaning. Regex Match asks whether an output matches a shape; Valid JSON asks whether it parses; word-overlap F1 asks how many words two texts share. None of them can tell you whether "low" should have been "urgent".

The compressed version: deterministic metrics check specific textual or structural properties; LLM Judge evaluates semantic agreement with the Expected Answer — and neither replaces looking at Case details.

Four practices follow: spotting the semantic gap in outputs that pass deterministic checks, comparing Valid JSON against LLM Judge on the same job, running LLM Judge on a paraphrased-FAQ job, and using Case details to explain what the judge caught that no deterministic check could.