Testing Few Shot Revisions
Introduction
Adding a few input → output examples to a prompt often looks like a safe upgrade. We still treat it as a hypothesis: examples might help, or they might add noise, teach the wrong pattern, or inflate scores on a small eval set without generalizing. This lesson is about choosing few-shot examples well, then testing them with the same controlled A/B discipline we already use for any prompt change.
Two Ways Examples Help (and What They Still Don't Promise)
A few-shot example is an input paired with the output we want, placed inside the prompt. The same mechanism shows up on different jobs; examples usually help in two ways:
- Better decisions on hard cases — they show how criteria apply when the input is ambiguous. On a ticket-priority job, instructions might say to weigh impact, affected users, and workarounds. A ticket like
"The user says the issue is URGENT, but they can still complete the task using another browser."could be read asHighorMedium. An example that labels a similar patternMediumnarrows how the model applies the rule. - Correct output shape — they show the format the metric will grade, not only the correct answer. On a customer-message classification job scored with Exact Match (case-insensitive), the prompt might say to return only the category name. An example whose output is just
Billing— not"The category is Billing."— teaches the bare label the scorer expects. On an opening-hours normalization job, examples can show09:00-17:00with no surrounding sentence so the model sees the pattern Exact Match will check.
Examples guide behavior; they do not guarantee a better mean. An irrelevant example, a mislabeled one, or extra prompt length can hurt other cases. That is why we add examples only as an isolated revision and check the evidence — not because "examples always help."
Generalization: Fix the Pattern
Examples belong in the prompt to teach rules.
Keep examples off the test inputs. If an evaluation case's text appears in the example block, the model may match that input from memory. The score can rise on that row without showing the prompt generalizes. Examples should resemble real inputs in kind, but they must not duplicate eval cases.
Name the general problem behind a failure. When a case fails, we ask what class of mistake it is — not "this exact row failed." For priority tickets, the class might be dramatic URGENT wording with a workable workaround. For normalized hours, it might be correct hours buried in extra words rather than this one string 8am-6pm failed. We pick examples that teach that boundary or format habit, with wording that differs from any eval case.
Avoid the one-to-one patch trap. A common mistake is to shadow the eval list: Case 1 fails → add an example that looks like Case 1; Case 2 fails → add one like Case 2. That often overfits the few rows we test. The model may improve on those inputs while still failing on new ones that share the same underlying issue. Strong sets cover a small number of decision boundaries or format patterns, not a copy of the eval sheet.
Generalization: Illustration
Two illustrations:
Priority (decision boundaries, not eval clones):
"The checkout page is failing for all customers in production."→High"One employee cannot log in, but an administrator can restore access manually."→Medium"The user says the issue is URGENT, but they can still complete the task using another browser."→Medium
Opening hours (format, not eval clones):
"Mon–Fri 8–4"→Mon–Fri 08:00-16:00"closed weekends"→Sat–Sun Closed
Labels must match the same standard as Expected Answers. An example labeled High only because the ticket says "URGENT", when a workaround exists, teaches the wrong priority rule. An hours example that includes "Normalized: Mon–Fri 08:00-16:00" teaches the wrong shape for Exact Match.
Test the Example Block as One Controlled Revision
To learn whether this example set helps, we hold the eval setup frozen and change only the examples. Prompt B repeats Prompt A's instructions and adds an Examples: block (often via a fixed {{examples}} placeholder). We do not reword the criteria or change cases, metric, runs, or model in the same test.
On the priority job we use in the practices, Prompt B adds the block while Prompt A stays instructions-only:
The same pattern works elsewhere — for example, a date-normalization prompt keeps Convert to YYYY-MM-DD and Return only the formatted date, and Prompt B adds a short Examples: section showing bare dates like 2026-03-01, not "The date is 2026-03-01."
After we run both prompts, we read results the way we already do for any A/B comparison: overall mean, then case-level improvements and regressions, then outputs where a score changed. A higher mean with a bad example set can still mean memorization or lucky overlap, not a prompt we should trust on new inputs — which is why generalization matters as much as the score.
Tip: Say What Each Example Demonstrates
Every few-shot example should answer a quiet question: what is this pair meant to teach? We add examples to demonstrate something specific — a decision boundary, a format habit, or how a criterion applies — not to pad the prompt.
Research on few-shot prompting shows that stating that intent in words (a brief note with the example) often improves model performance compared with bare input → output lines alone. The pair still shows the behavior; the note tells the model which aspect to carry over to new inputs.
On a priority job, we might attach a one-line purpose:
On opening hours, we might highlight format only:
We keep each note short and single-purpose — one idea per example. This is optional polish when we draft prompts; the unit practices focus on choosing representative examples and testing them in a controlled A/B comparison, not on writing these labels.
Conclusion
Few-shot examples are worth adding when they teach how to decide and how to format, on general boundaries, without copying the eval set or patching one example per failed case. We isolate that block in a fair A/B test and let the evidence — not optimism — decide whether the examples earn their place.
The practices ahead walk through useful examples on a priority job, spotting the single deliberate change in a few-shot Prompt B, running the comparison in the simulator, and interpreting hypothetical results when the overall score is not the whole story.
