Project 09 — A Tiny Eval Harness

Enforces: building an eval harness — a dataset, two grading methods (code-based and LLM-as-judge), an A/B of two prompt versions, and a Batch API run (from prompt-evaluations) Surface: code · Time: ~90 min · Difficulty: 🔴 stretch

Why this project

Without evals, prompt engineering is vibes. An eval harness turns "this prompt feels better" into "version B scores 0.91 vs 0.78 on 20 cases." Once you can measure, every later change — a new model, a tweaked system prompt, an added example — becomes a decision instead of a guess.

What you'll build

A small harness with a dataset of inputs + expected answers, two graders (code-based exact/regex and LLM-as-judge), an A/B comparison of two prompt versions, with the model run executed through the Batch API.

Steps

Build a dataset — 15–20 cases for one narrow task (e.g. classify a sentence as positive / negative / neutral, or extract a date). Each case is {"id", "input", "expected"}. Include a few deliberately hard/ambiguous cases — that's where prompt versions diverge.
Write two prompt versions — PROMPT_A (a plain instruction) and PROMPT_B (same task, but with role + one example + an explicit output format). You're testing whether structure helps on this task.

Run with the Batch API — for each (prompt version × case), submit one request. The Batch API processes them asynchronously at half price — ideal for eval sweeps. Key each request by a custom_id that encodes version + case id, because results come back unordered:

from anthropic.types.message_create_params import MessageCreateParamsNonStreaming
from anthropic.types.messages.batch_create_params import Request

requests = []
for version, system in [("A", PROMPT_A), ("B", PROMPT_B)]:
    for case in dataset:
        requests.append(Request(
            custom_id=f"{version}::{case['id']}",
            params=MessageCreateParamsNonStreaming(
                model="claude-opus-4-8", max_tokens=256,
                system=system,
                messages=[{"role": "user", "content": case["input"]}],
            ),
        ))
batch = client.messages.batches.create(requests=requests)

Poll, then collect by custom_id — wait for processing_status == "ended", then stream results and key them by custom_id (never by position):

import time
while client.messages.batches.retrieve(batch.id).processing_status != "ended":
    time.sleep(10)
outputs = {}
for r in client.messages.batches.results(batch.id):
    if r.result.type == "succeeded":
        text = next((b.text for b in r.result.message.content if b.type == "text"), "")
        outputs[r.custom_id] = text

Grade with code (exact/regex) — the cheap, deterministic grader. Normalize and match against expected:

import re
def grade_code(output, expected):
    return bool(re.fullmatch(re.escape(expected), output.strip(), re.IGNORECASE))

Grade with LLM-as-judge — for cases where exact match is too brittle (paraphrases, partial credit), ask a separate model call to score the answer against the expected, returning a forced-tool verdict ({"correct": bool, "reason": str}). Use this as a second signal, not a replacement for the code grader.

judge_tool = {"name": "verdict", "description": "Judge an answer against the expected.",
              "input_schema": {"type": "object",
                "properties": {"correct": {"type": "boolean"}, "reason": {"type": "string"}},
                "required": ["correct", "reason"]}}
def grade_judge(output, expected):
    r = client.messages.create(model="claude-opus-4-8", max_tokens=256,
        tools=[judge_tool], tool_choice={"type": "tool", "name": "verdict"},
        messages=[{"role": "user",
            "content": f"Expected: {expected}\nGot: {output}\nIs the answer correct?"}])
    return next(b.input for b in r.content if b.type == "tool_use")["correct"]

Score and compare — compute each version's pass rate under each grader. Report a small table: version × grader → score. Note where the two graders disagree — those cases reveal both prompt weaknesses and grader blind spots.

Acceptance criteria — you're done when

You have a 15–20 case dataset of {input, expected} including a few hard cases.
Two prompt versions run over every case via the Batch API, keyed by custom_id.
You collect results by custom_id (not position) after polling to ended.
A code-based exact/regex grader and an LLM-as-judge grader both score every output.
You produced a version × grader score table and identified at least one grader disagreement.
You can state which prompt version won and on what evidence.
You journaled what the eval revealed that your intuition missed in learning-journal-template.

Stretch goals

Add prompt caching to the shared system prompt across batch requests and confirm the cache hit.
Add a third grader (semantic similarity) and see which grader best matches your own human judgment.
Re-run the winning prompt on a faster/cheaper model and decide if the quality drop is worth the savings.

Self-assessment rubric

Level	Signal
🟢 Got it	You pick prompts by score, run sweeps on the Batch API, and know when to trust each grader.
🟡 Almost	The harness runs but you lean on one grader or compare versions by eyeballing outputs.
🔴 Revisit	Results feel arbitrary or the batch keying breaks; re-watch prompt-evaluations.