▲ Claude Academy
Sign in

Vault note

Course: Prompt Evaluations

Mirrors: Anthropic — Prompt Evaluations · https://anthropic.skilljar.com/prompt-evaluations Audience: Developers and practitioners who want to ship prompts with confidence. · Time: ~80 min + project Prereqs: real-world-prompting; comfort calling the API. · Backing notes: structured-output, batch-api, cca-domain-4-prompting Project: p09-eval-harness

"It looks good" is not a release criterion. This course teaches you to measure a prompt: build an eval dataset, grade outputs with the right method, develop prompts test-first, track metrics and regressions, and trade cost against latency — so you can change a prompt and know whether you made it better.

Learning objectives

After this course you can:

  • Explain why you evaluate a prompt before shipping it.
  • Build an eval dataset of inputs paired with expected outputs / grading criteria.
  • Choose a grading method: exact/code-based, LLM-as-judge, or human.
  • Develop prompts test-first (TDD for prompts).
  • Define metrics, run a regression suite, and iterate against the score.
  • Reason about the cost/latency tradeoffs of running evals at scale.

Module 1 — Why and what to evaluate

🎞 Frame 1 · Why eval before you ship · ⏱ ~3 min

🎬 Scene — A prompt tweak that "looked fine" silently regresses 12% of real cases in production; an eval would have caught it pre-merge.

🧠 Concept — Without evals, prompt changes are guesses you can't grade. An eval turns "feels better" into a number, catches regressions before users do, and lets you compare prompts and models objectively. (Cert context: cca-domain-4-prompting.)

🖼 On screen

No eval:   change prompt → eyeball one output → ship → discover regressions in prod
With eval: change prompt → run 50 graded cases → see score move → ship with evidence

✅ Checkpoint — Name one failure an eval catches that eyeballing one output won't. (A regression on cases you didn't happen to look at.)

🎞 Frame 2 · Build an eval dataset · ⏱ ~4 min

🎬 Scene — A spreadsheet fills with rows: input, expected output (or grading criteria), and a category tag.

🧠 Concept — An eval dataset is inputs paired with expected outputs or grading criteria. Cover typical cases, edge cases, and known past failures. 20–50 well-chosen cases beat 500 near-duplicates.

🖼 On screen

[
  {"id": "t01", "input": "I want to cancel", "expected": "cancellation", "tag": "typical"},
  {"id": "t02", "input": "charge looks wrong", "expected": "billing", "tag": "typical"},
  {"id": "t03", "input": "ASDF ???",          "expected": "spam",    "tag": "edge"},
  {"id": "t04", "input": "cancel AND refund",  "expected": "needs_human", "tag": "edge"}
]

⚠️ Gotcha — Every production bug should become a new eval row, so it can never silently return.

✅ Checkpoint — What three kinds of cases should an eval set deliberately include? (Typical, edge, and past-failure cases.)

Module 2 — Grading methods

🎞 Frame 3 · Code-based / exact grading · ⏱ ~3 min

🎬 Scene — Outputs run through an assert: exact match, regex, or JSON-schema validation — instant pass/fail.

🧠 Concept — When the right answer is deterministic (a label, a number, a schema), grade with code: exact match, regex, set membership, or schema validation. Cheapest, fastest, fully reproducible. (Deeper: structured-output.)

🖼 On screen

def grade(case, output):
    if case["tag"] == "schema":
        return validates_against_schema(output)      # jsonschema / pydantic
    return output.strip().lower() == case["expected"] # exact match

✅ Checkpoint — Which grading method fits a fixed-label classifier? (Code-based exact match.)

🎞 Frame 4 · LLM-as-judge · ⏱ ~4 min

🎬 Scene — A second Claude call scores a free-text answer against a rubric, returning a number and a justification.

🧠 Concept — For open-ended outputs (summaries, replies) where exact match fails, use LLM-as-judge: prompt a model with the input, the output, and a clear rubric; force a structured score. Use a strong model (Sonnet/Opus) as the judge. (Force the score with a tool — structured-output.)

🖼 On screen

judge_tool = {
    "name": "score", "description": "Score the answer against the rubric.",
    "input_schema": {"type": "object", "properties": {
        "score": {"type": "integer", "description": "1–5"},
        "reason": {"type": "string"}}, "required": ["score", "reason"]}}

# system: "You are a strict grader. Rubric: 5=fully correct & grounded ... 1=wrong/hallucinated."
# user: <question>...</question><answer>...</answer>
# tool_choice forces the `score` tool → reliable numeric output.

⚠️ Gotcha — Judges are biased toward longer/verbose answers and can drift. Pin the judge model, give a concrete rubric with anchored examples, and spot-check the judge against human labels.

✅ Checkpoint — Why force the judge's output through a tool? (To get a reliable numeric score you can aggregate, not prose.)

🎞 Frame 5 · Human grading · ⏱ ~3 min

🎬 Scene — A small sample of outputs is hand-labelled by a person; their labels calibrate the automated graders.

🧠 Concept — Human grading is the gold standard but doesn't scale. Use it for a calibration sample, for subtle quality (tone, helpfulness), and to validate that your LLM-judge agrees with humans.

🖼 On screen

Pyramid of grading effort:
  Human      — small, gold-standard, calibrates the rest
  LLM-judge  — medium volume, open-ended outputs
  Code-based — high volume, deterministic outputs

✅ Checkpoint — What do you use human labels for if they don't scale? (To calibrate/validate code and LLM-judge graders, and to grade subtle quality.)

Module 3 — Test-driven prompting, metrics, and tradeoffs

🎞 Frame 6 · Test-driven prompt development · ⏱ ~4 min

🎬 Scene — The eval set is written before the prompt; the prompt is then tuned until the score crosses a target.

🧠 Concept — Flip the order: write the eval first, then develop the prompt until it passes. The eval becomes your definition of "done" and your regression net.

🖼 On screen

1. Write eval cases (inputs + expected/criteria).
2. Set a target (e.g. ≥90% pass on typical, ≥75% on edge).
3. Draft prompt → run eval → read failures.
4. Change ONE thing → re-run → keep wins.
5. Crossed target? Ship. Keep the eval as the regression suite.

🔗 This is the disciplined version of the iterate loop from real-world-prompting.

✅ Checkpoint — What plays the role of "the test" in test-driven prompting? (The eval dataset + its pass target.)

🎞 Frame 7 · Metrics & regression suites · ⏱ ~4 min

🎬 Scene — A run report: overall accuracy, per-tag breakdown, and a diff against the last run flagging two newly-failing cases.

🧠 Concept — Track the right metrics (accuracy/precision/recall for classification; rubric score for open-ended) and run the suite on every change so a fix in one place can't quietly break another (a regression).

🖼 On screen

RUN 2026-06-25   prompt v7   model claude-sonnet-4-6
  overall   88%  (▲ +3)
  typical   96%  (=)
  edge      71%  (▲ +9)
  REGRESSION: t02 typical now FAILS (was PASS in v6)  ← block the merge

⚠️ Gotcha — A higher overall score can still hide a regression on an important sub-slice. Always read the per-tag breakdown and the diff vs. last run, not just the headline number.

✅ Checkpoint — Why report per-tag scores and a diff, not just overall accuracy? (Overall can rise while a critical slice regresses.)

🎞 Frame 8 · Iterating against the eval · ⏱ ~3 min

🎬 Scene — Failing cases are clustered by failure mode; a single targeted prompt change lifts a whole cluster.

🧠 Concept — Let the failures drive iteration. Cluster them by mode (hallucination, wrong format, missed edge), fix the biggest cluster first, re-run. The eval tells you if the fix generalized or just patched one case.

🖼 On screen

Failures by mode:
  hallucination  ████████ 8   → ground in context + "not found" out
  wrong format   ███ 3        → force schema + prefill
  missed edge    ██ 2         → add 2 few-shot edge examples
Fix the biggest cluster → re-run → confirm the cluster cleared.

✅ Checkpoint — Why fix by cluster rather than case-by-case? (One change clears many cases and generalizes; per-case patches overfit.)

🎞 Frame 9 · Cost & latency of evals · ⏱ ~3 min

🎬 Scene — A 2,000-case eval submitted as one overnight job at half price instead of 2,000 live calls.

🧠 Concept — Big eval suites are exactly the offline, high-volume workload the Batch API was built for — 50% cheaper, latency irrelevant. Use a cheaper model for the candidate where the task allows, and reserve the strong model for the judge. (Deeper: batch-api.)

🖼 On screen

Eval at scale:
  Run the 2,000 candidate calls via the Batch API   → 50% off, overnight
  Judge open-ended outputs with Sonnet/Opus (also batchable)
  Code-grade everything deterministic for free
Tradeoff: more cases & a stronger judge = more cost; batch + code-grading claw it back.

✅ Checkpoint — Which API runs a 2,000-case eval cheapest, and why is latency a non-issue? (Batch API — 50% off; evals are offline, so 24h turnaround is fine.)

🛠 Project

Complete p09-eval-harness — Build an Eval Harness: assemble a 20–30 case dataset (typical + edge + past-failure) for a real task, implement code-based grading for the deterministic parts and an LLM-as-judge with a forced-tool rubric for the open-ended parts, run a baseline, then iterate the prompt against the score and produce a regression report with a per-tag breakdown and a diff vs. baseline.

🧪 Self-check quiz

  1. What does an eval give you that eyeballing a single output does not?
  2. What three kinds of cases belong in an eval dataset?
  3. Match each grading method to a use: fixed labels / open-ended quality / gold calibration.
  4. Why force the LLM-judge's score through a tool?
  5. Name two biases of LLM-as-judge and a mitigation for each.
  6. In test-driven prompting, what is written first and what defines "done"?
  7. Why read per-tag scores and a run-to-run diff instead of just overall accuracy?
  8. Which API runs a large eval cheapest, and why is its latency acceptable?
<details><summary>Answers</summary>
  1. A score over many cases that catches regressions you didn't look at. 2. Typical, edge, and past-failure cases. 3. Code-based exact match / LLM-as-judge / human grading. 4. To get a reliable, aggregatable numeric score instead of prose. 5. Verbosity/length bias and drift; mitigate with a concrete anchored rubric, a pinned judge model, and human spot-checks. 6. The eval dataset + pass target is written first and defines "done". 7. Overall can rise while a critical sub-slice regresses; the diff flags newly-failing cases. 8. The Batch API (50% off); evals are offline so a 24h turnaround is fine.
</details>

🎓 Certificate criteria

You've "passed" Prompt Evaluations when you can:

  • Explain why you eval before shipping and what a regression is.
  • Build an eval dataset covering typical, edge, and past-failure cases.
  • Pick and implement the right grading method (code / LLM-judge / human) per output type.
  • Develop a prompt test-first and produce a regression report with a per-tag breakdown.
  • Complete p09-eval-harness and journal one regression your eval caught.

Tick this course off in progress and record the date you earned Anthropic's official certificate.

🔗 Sources & deeper notes