Foundations of Prompt Evaluation

A prompt is the text you send to a language model to get work done — the instructions, plus any input the model should act on. "Summarize this ticket in two sentences" is a prompt. So is a page of rules telling a model how to draft customer replies, which tone to use, and what it must never say. In a product, that text is not a one-off message you type into a chat window; it is a component that runs thousands of times a day on inputs you have never seen.

Most prompt work ends the same way: you write that text, read a few outputs, they look right, and you move on. This lesson is about what happens next — because "it looked right" is not evidence, and it tends to fail in ways you did not test for. Here is what that looks like in practice.

A support team ships an assistant that drafts replies to billing questions. The prompt was tested by hand: someone typed three questions, read the answers, thought "that sounds great," and shipped it.

Monday morning, the complaints start. The assistant is confident and fluent — and it invents a refund window that does not exist. It answers in English when the customer writes in German. When a customer asks about two charges in one message, it addresses only the first. None of this showed up in the three hand-typed questions, because those three questions happened to be short, English, and single-topic.

Nothing was wrong with the writing of that prompt. What was missing was a way to check it: a set of cases that reflect real traffic, a stated bar for a passing answer, and a repeatable run that produces evidence instead of an impression.

That gap is what this course closes. In this first unit, you will learn what prompt evaluation actually is, why an evaluation needs a decision behind it, how to turn "good" into something observable, and where these checks pay off in real work.

Prompt evaluation is the practice of checking a prompt against cases you chose in advance, using a stated definition of a passing answer, under conditions you can repeat.

Three details in that sentence do the work:

• Chosen in advance. You decide what the prompt must handle before you look at the outputs, so you are not grading the cases the prompt happens to be good at.
• A stated bar. "Good" is written down as something visible in the output, so two people inspecting the same answer agree on whether it passed.
• Repeatable. The same prompt, cases, and settings can be run again — after an edit, or a week later — and the results stay comparable.

This matters because language models are non-deterministic and prompts are fragile. The same prompt can answer well on Tuesday and drift on Wednesday; a one-word edit meant to fix a formatting nit can quietly break a different case. Reading a few outputs tells you how a prompt behaves on those outputs. Evaluation tells you how it behaves on the work you care about, and it gives you something you can show a teammate or a stakeholder instead of asking them to trust your taste.

Your work as a Prompt Engineer does not end when a prompt sounds polished. You need a repeatable way to tell whether it helps users, works across the cases that matter, and is ready for a product decision. This course turns vague claims about quality into evidence you can inspect and discuss.

By the end of this course, you'll be able to:

• State the decision an evaluation should support and translate "good" into observable criteria.
• Build tiny case sets with expected answers and choose a metric that fits the answer shape.
• Run repeated checks under fixed conditions and interpret variation across runs and cases.
• Use Prompt Eval scores and consistency to recommend ship, revise, or gather more evidence.

You will build those skills across the course. This unit starts the foundation: name the decision, define observable quality, then meet the tool you will use to check both. Later units add metrics, multi-run evidence, and recommendations.

An evaluation is not a general opinion about whether a prompt feels polished. It is a planned check that produces evidence for a decision, such as whether to keep a prompt, revise it, compare it with another version, or gather more information.

Start by asking, "What decision will this evaluation support?" "Is this prompt good?" is too broad. A better question is, "Should we use this prompt for this specific job?" The job gives you the context for judging the output.

For a capital-city lookup, the decision might be whether the prompt reliably gives the correct city in the format users need. For an open brainstorming prompt, the decision might be whether it produces useful options without ignoring the user's constraints. For the billing assistant above, the decision is narrower and more pointed: can this prompt draft a first reply that a human agent can send with light editing?

Without a clear decision, evaluation often devolves into "vibe checking"—subjective tinkering that lacks a clear finish line. A well-defined decision links your test results to a specific action: deciding to ship the prompt, identifying exactly which edge cases require a logic change, or choosing between two competing versions based on performance data rather than personal preference.

Once you know the decision, describe quality in terms of what you can see in a model output. A criterion is a condition an output can meet or miss. It turns a vague goal such as "be helpful" into something that can be checked consistently.

For a short factual answer, requirements are not interchangeable. "Return the exact city name only" means the output should match the expected city string with nothing added. "The city name may appear inside a sentence" allows surrounding text, as long as the required city is present. "Give any helpful geography fact" is broader still, and a simple text check may not be enough.

The same move works on messier jobs. "Be helpful to the customer" is not checkable. "Replies in the language of the incoming message," "names a refund window only if it appears in the policy text provided," and "addresses every charge the customer mentions" are all conditions you can point at in an output and mark pass or fail.

A check that is too strict may reject a correct answer in an acceptable format. A check that is too loose may approve text that contains a familiar word but does not answer the question. Your first job is not a fancy scoring method. It is to state the quality bar clearly enough that another person could inspect the same output and reach a similar conclusion.

Prompt evaluation is not reserved for trivia lookups. It earns its keep anywhere a prompt sits in a product or a workflow:

• Ticket triage. A prompt labels incoming support tickets as billing, bug, or account. The decision: is it accurate enough to route tickets automatically, or should it only suggest a label to a human?
• Structured extraction. A prompt pulls vendor, date, and total from invoice text into JSON. The decision: does it produce valid, correctly filled fields often enough to feed the downstream system?
• Grounded Q&A over documents. A prompt answers policy questions using retrieved passages. The decision: does it stay inside the provided text instead of filling gaps from memory?
• Summarization. A prompt condenses a long call transcript for a manager. The decision: does it keep the commitments and next steps, not just the pleasantries?
• Rewriting and tone. A prompt turns internal engineering notes into customer-facing release notes. The decision: does it drop internal jargon and keep the facts intact?
• Comparing two prompt versions. You rewrote the instructions to fix one annoying case. The decision: did the new version actually improve things overall, or trade one failure for another?

In every one of these, the same three questions apply: what decision am I supporting, what does a passing output look like, and which cases must the prompt handle before I trust it?

Throughout this course, you will use the Prompt Evaluation Simulator to check prompts with evidence. It keeps a prompt, test cases, expected answers, and a scoring setup in one place so you can move from guessing whether a prompt works to inspecting results you can discuss.

The simulator is not a magic judge. Its value depends on whether the setup reflects the job and the decision you named.

Use the numbered map to spot each control. You will learn what each one is for across the course.

Some parts of this screen may look unfamiliar at first — for example metric, runs, or how scores appear in Results. That is expected. You do not need to master every control in this unit. Across the course you will learn what each piece is for by using it on a real job.

For now, use the annotated map to spot four ideas:

• the prompt you are checking
• the cases (questions) it should handle
• an optional expected answer used for scoring later
• the controls that keep a check repeatable (such as metric and runs)

In the practices, you will get the chance to use this simulator to understand how it works.

The billing assistant did not fail because someone wrote a bad prompt. It failed because "it sounded great" was the only evidence anyone had.

Three ideas from this unit replace that:

1. Evaluation serves a decision. Before you test anything, name the action the result will support — ship, revise, compare, or investigate further.
2. Quality has to be observable. Rewrite "be helpful" as conditions you can point at in an output, so another person inspecting it would reach the same verdict.
3. Evidence lives in a setup you can rerun. A prompt, chosen cases, expected answers, and fixed settings make today's result comparable to tomorrow's.

In the practices, you will open a seeded capital-city job in the simulator — a starter prompt, a few cases such as France → Paris, and a metric waiting for a later run — and read that setup through these three lenses. Nobody ships an LLM to look up capitals, and that is exactly why it is a good first job: the correct answer is never in dispute, so all of your attention goes to the evaluation mechanics rather than to arguing about whether an answer was good.

Once those mechanics are second nature, the course points them at jobs with real ambiguity and adds the pieces that turn a setup into numbers you can defend: metrics matched to the answer shape, repeated runs that expose inconsistency, and a recommendation you can put in front of a stakeholder.