Understanding Format Evaluators

So far, checking a structured output has meant opening it up and reading it yourself — comparing keys, checking values, deciding by hand whether the format, structure, and semantics all hold up. That works, but it doesn't scale well, and some properties of an output are actually simple enough for an automatic check to confirm reliably. If an order ID always follows the same shape, or a reply is supposed to be JSON, you shouldn't have to eyeball every single case to know whether that narrow property held.

This lesson introduces two evaluators built for exactly that: Regex Match, which scores a case by whether its output follows a predictable text pattern, and Valid JSON, which scores a case by whether its output can be parsed as JSON at all. Both are fast, deterministic, and genuinely useful — and both answer a much narrower question than "is this correct?" You'll learn what each one actually tests, see both evaluators in action, and practice the habit that keeps you from over-trusting a pass: before accepting a score, ask exactly what question that evaluator was asking.

Before the first evaluator makes sense, you need one small idea: a regex (short for "regular expression") is a compact template that describes the shape of a piece of text rather than its exact content.

Think of how you'd describe an order ID to a coworker: "the letters ORD, then a dash, then five digits." That description doesn't name any one ID — it covers all of them. A regex is that same description, written in a form an evaluator can check automatically. You don't need any coding background to read one — just enough symbols to know what will be accepted:

• ^ — means "start here." The match must begin at this point.
• $ — means "end here." The match must stop at this point.
• \d — means "a digit" (any single number 0–9).
• {5} — means "exactly 5 of the previous thing."

Regex supports many other special symbols beyond these, but this lesson only uses ^, $, \d, and {5}. In the pattern used here, anything that isn't one of those symbols is just literal text that has to appear exactly as written — which is why ORD- means the characters O, R, D, -, in that order.

So you read ^ORD-\d{5}$ left to right, as a template: start here, the literal text ORD-, then exactly five digits, then end. Under that pattern, ORD-12345 matches. ORD-123 does not, because it only has three digits where five are required. Order 12345 does not either, because it doesn't start with the literal text ORD-.

You won't be writing patterns from scratch in this course — the patterns you need will be given to you. What matters is being able to read one well enough to know precisely what a pass from it does and doesn't prove.

Regex Match takes a pattern like that one and scores each case by testing the output against it: pass when the shape matches, fail when it doesn't.

The pattern goes into a field you've already used: Expected Answer. Instead of typing a ground-truth string in Expected Answer, you provide the regex pattern that the output should match. When the evaluation runs, each case's output is checked against that pattern rather than compared as text — so what you're describing is a shape, not one specific correct value.

That's also the limit of what a pass proves. A passing case means the output's shape satisfies the pattern — nothing more. If the pattern is ^ORD-\d{5}$ and the model outputs ORD-99999, that case scores 1, even if ORD-99999 isn't the correct order ID for that particular input. Regex Match only checks the conditions you actually encoded into the pattern. It has no way to know what the "right" ID is — it only knows what a correctly-shaped ID looks like.

Picture running this on an order-ID job with ^ORD-\d{5}$ as the Expected Answer, across 4 cases. Outputs ORD-88213, ORD-40410, and ORD-00027 each match the pattern and score 1. An output like Order #5521 fails, since it doesn't start with ORD-, scoring 0. That gives a mean of 3 out of 4, or 0.75 — and as you saw back in Course 1, a mean like that is a signal to open Case details on the failing case, not a number to accept at face value.

Regex Match is a strong fit when an output is expected to follow one predictable shape, like an ID, a code, or a formatted date. But once outputs become full JSON objects with multiple keys and varying values, writing a single pattern to describe the whole structure gets impractical fast. That calls for a different kind of check.

Valid JSON scores each case on a single, narrow question: can this entire output be read as JSON? That means checking that brackets are balanced, quotes are properly closed, commas are in the right places, and nesting is well-formed.

Unlike Regex Match, Valid JSON doesn't need an Expected Answer at all — there's no pattern or ground truth to type in. You select Valid JSON as the metric and run it. The evaluator attempts to parse each output, and the case scores 1 if the parse succeeds, or 0 if it fails.

So a passing case means the output is syntactically well-formed JSON. It does not mean the required keys are present, that a particular schema was followed, or that any value inside is correct. Those are different questions, and Valid JSON isn't built to answer them.

Imagine a ticket-triage job, where the model is asked to reply with a JSON object describing a support ticket. Running Valid JSON over 4 cases, an output like {"priority":"high","team":"billing"} parses cleanly and scores 1. An output like {"priority":"high","team":"billing" — missing its closing brace — fails to parse and scores 0. If three of four cases parse successfully, the mean comes out to 3/4, or 0.75 again — the same arithmetic as the regex example, but this time the failing case is a broken parse, not a wrong-shaped ID.

Both evaluators can return a clean, confident-looking pass. But a pass from either one still leaves an important question completely open.

An output can be perfectly parseable JSON and still contain a nonsense or incorrect value. Suppose a priority-triage job clearly calls for "urgent" given the input, but the model instead outputs {"priority":"banana"}. That string parses without any error — the brackets, quotes, and commas are all in order — so Valid JSON scores that case 1. But "banana" isn't a real priority level, and it's wrong for the job. The format check passed while the actual requirement failed.

The same gap applies to Regex Match. An order ID that matches ^ORD-\d{5}$ might still be the wrong order ID for that case. The pattern only confirms shape; it never checks whether that specific value belongs in that specific case.

It's worth stepping back here: the evaluators in this lesson are deliberately simple examples, chosen to make one narrow question easy to see. Nothing stops you from building something richer. A Valid JSON check could be extended to also confirm that the required keys are present, sit in the right place, and hold sensible values — at which point it's answering more than just "does this parse?" The broader point is that the metrics you meet in this course are examples, not a fixed menu. In a real experiment, you're free to design whatever comparison between the model's answer and the ground truth actually fits the job in front of you.

With the evaluators covered so far, semantic correctness — whether the content is actually right — is still not something either check measures automatically. For now, the way to catch it is to open Case details and read the actual output next to the input yourself. That's not a permanent state of affairs — a later unit introduces an evaluator built specifically to judge meaning — but it's the tool available right now, and it's a reliable one as long as you remember to use it.

The shift here is in what you expect a passing score to prove. Regex Match and Valid JSON are both fast, deterministic checks — but each one is only ever answering the question it was built to ask, and neither one was built to judge whether the content is actually right.

Hold onto the pattern: Regex Match asks "does it match this shape?" Valid JSON asks "can this be parsed?" Neither asks "is this actually correct?" Every time you see a pass from either metric, that's the question to ask next.

You'll see both evaluators in action: Regex Match against an order-ID job to see pattern-shape evidence firsthand, Valid JSON against a ticket-triage job to see parse-validity evidence, and then a case where a reply parses just fine but still gets the priority wrong — putting into words exactly why a passing format check and a failing job requirement can sit side by side in the same case.