Project 08 — Q&A over Docs with Prompt Caching + RAG

Enforces: prompt caching with cache_control breakpoints, reusing the cache across many questions, and basic retrieval to select relevant chunks (from building-with-the-claude-api) Surface: code · Time: ~90 min · Difficulty: 🔴 stretch

Why this project

Two ideas power most document-Q&A systems: caching (don't re-pay to process the same context on every question) and retrieval (don't stuff everything into context — fetch what's relevant). This project builds a small Q&A-over-docs that does both, and — crucially — makes you verify the cache actually hit.

What you'll build

A Q&A tool that loads a handful of documents into context with a cache_control breakpoint, asks several questions while reusing the cache, compares cost/latency against an uncached baseline, then adds basic retrieval to select only relevant chunks.

Steps

Load a few documents — gather 3–6 documents totaling well over the cacheable minimum (≥4096 tokens for Opus). Concatenate them into one stable context string. Count tokens with client.messages.count_tokens(...) to confirm you're over the threshold.

Place a cache breakpoint — caching is a prefix match: stable content first, the varying question last. Put cache_control on the last stable block (the documents), and keep the per-question text after it:

def ask(question):
    return client.messages.create(
        model="claude-opus-4-8", max_tokens=1024,
        system=[{
            "type": "text",
            "text": f"Answer only from these documents.\n\n{DOCS}",
            "cache_control": {"type": "ephemeral"},   # caches the doc prefix
        }],
        messages=[{"role": "user", "content": question}],  # varies — after the breakpoint
    )

Ask multiple questions and verify reuse — fire 4–5 questions in sequence. Inspect usage on each: the first writes the cache (cache_creation_input_tokens > 0), later ones read it (cache_read_input_tokens > 0). If cache_read_input_tokens stays 0, a silent invalidator is at work — a timestamp or UUID in the prefix, non-deterministic key order, or content shorter than the minimum.
```
r = ask("What is the refund policy?")
print(r.usage.cache_creation_input_tokens, r.usage.cache_read_input_tokens, r.usage.input_tokens)
```
Compare cached vs uncached — run the same questions once without the breakpoint and record input-token cost and latency per question. Cache reads cost ~0.1× of base input; the write costs ~1.25×. Tally the savings across the batch.
Add basic retrieval — instead of always sending all docs, split them into chunks, score each chunk against the question (keyword overlap is fine for v1; embeddings if you want), and put only the top-k chunks into context. Now the prompt is smaller and still answerable.
Reason about the tension — retrieval shrinks the prefix but changes it per question, which can defeat caching. Note the design tradeoff: cache a large stable shared prefix when many questions reuse it; retrieve a small varying prefix when each question needs different chunks. State which fits your docs and why.

Acceptance criteria — you're done when

Your document prefix exceeds the cacheable minimum (verified with token counting).
A cache_control breakpoint sits on the stable docs, with the per-question text after it.
You showed cache_creation_input_tokens > 0 on the first call and cache_read_input_tokens > 0 on later calls.
You compared cached vs uncached input-token cost and latency across the question batch.
You added retrieval that selects top-k relevant chunks instead of sending everything.
You wrote down when caching wins vs when retrieval wins for your docs.
You journaled one silent cache-invalidator you hit or avoided in learning-journal-template.

Stretch goals

Pre-warm the cache with a max_tokens=0 request at startup and measure the first-question latency drop.
Try the 1-hour TTL ({"type": "ephemeral", "ttl": "1h"}) and reason about the doubled write cost vs reuse.
Add citations (citations: {enabled: True} on document blocks) so answers point back to source spans.

Self-assessment rubric

Level	Signal
🟢 Got it	You can prove a cache hit from `usage` and explain the caching-vs-retrieval tradeoff for your docs.
🟡 Almost	Caching works but `cache_read_input_tokens` was zero until you hunted down an invalidator.
🔴 Revisit	The cache never hits and you're not sure why; re-read the prompt-caching material in building-with-the-claude-api.