Vault / wiki/301/practice/ccarp/domain-4-evaluation-testing-optimization.md
updated 2026-07-16Practice — CCAR-P Domain 4: Evaluation, Testing & Optimization (16%)
16 scenario-based MCQs. Answer key + explanations at the bottom.
Q1
You are building the first eval dataset for a customer-support classifier that routes messages into four intents. The team has 10,000 labeled production transcripts available. Which dataset composition best supports ongoing prompt iteration?
A. 200 synthetic examples generated by Claude so every intent class is perfectly balanced B. The 500 most recent production messages, taken as-is to reflect current traffic C. A production sample that covers typical, ambiguous, and refusal/edge cases for each intent D. The 300 hardest cases the team can find, since easy cases waste evaluation budget
Q2
A team has rewritten the system prompt for its extraction pipeline and wants to know whether v2 beats v1. Which procedure is the best practice?
A. Run v1 and v2 side-by-side on the same eval set and compare results per input class B. Deploy v2 to production and compare this week's complaint volume against last week's C. Run v2 on a freshly drawn eval set so the results are not biased by v1's test data D. Have a senior reviewer read ten outputs from each version and pick the stronger one
Q3
A RAG-based policy assistant confidently answers questions about coverage limits that appear nowhere in the retrieved chunks — the numbers it cites do not exist in the corpus. Which change most directly addresses the failure?
A. Upgrade the generation model from Sonnet to Opus to get stronger reasoning B. Add grounding instructions: answer only from <context>, and say so when the answer is absent C. Increase retrieval top-K so more chunks reach the model on every query D. Reduce max_tokens so the model has less room to elaborate beyond the sources
Q4
Your eval shows a compliance-disclaimer instruction is followed only 60% of the time. The system prompt is 3,200 tokens with roughly 40 same-priority rules; every other metric passes. The team proposes upgrading to a larger model. What should you conclude?
A. Model mismatch — a model with stronger instruction-following is needed to track this many rules B. Hallucination — add retrieval grounding so the disclaimer text is quoted from a source document C. The eval set is too small to measure instruction-following reliably, so gather more data first D. Prompt dilution — restate the critical rule at the top and end, or restructure, before changing models
Q5
Inference costs for a support assistant tripled in week three. Usage logs show high cache_creation_input_tokens on nearly every request — including messages sent seconds apart within the same session — and cache_read_input_tokens near zero. What is the most likely cause?
A. The cached prefix changes per request — e.g., a timestamp or user ID sits before the cache breakpoint B. Conversation histories have grown long and need compaction with a sliding window strategy C. The 5-minute cache TTL is expiring between each user's consecutive messages in a session D. The team exceeded the four-breakpoint limit per request, which silently disables all caching
Q6
A team caches a 40KB reference document in the system prompt of an internal tool that receives roughly one request every 20 minutes. Billing shows the caching change actually increased cost. Why?
A. Cache reads are billed at 50% of input cost, and this volume never amortizes the setup B. Documents of that size exceed what a single cache breakpoint is allowed to hold C. Reference documents belong in the user turn, where cached content bills at a lower rate D. The 5-minute TTL lapses between requests, so every call pays the ~125% cache-write premium
Q7
You maintain a nightly regression eval of 5,000 test cases and want to compare Haiku and Sonnet on identical prompts while minimizing spend. Which setup is best aligned?
A. Two synchronous scripts, one per model, executed in parallel overnight from a scheduler B. One Message Batches job mixing both models, with custom_id mapping each result to its test case C. A streaming pipeline so results arrive token-by-token and failures surface early in the run D. An orchestrator agent that loops through the cases, calls each model, and records verdicts
Q8
An invoice-extraction pipeline outputs JSON with vendor, date, totals, and a source pointer per field. You need an evaluation framework the audit team will accept. Which mix of methods is most appropriate?
A. An LLM judge scoring every output holistically on a 1–10 quality scale each night B. Human reviewers reading every extraction against its source document for full audit coverage C. Programmatic schema and field checks, plus human spot-checks of sampled outputs via provenance D. Exact string comparison of each full JSON output against a golden reference file
Q9
A team uses an LLM judge to score chatbot answers for "helpfulness" with no rubric. Scores swing widely between runs, and prompt changes chased from those scores show no consistent gains. What is the root cause?
A. The judge has no clear evaluation criteria, so its feedback is noise driving changes in random directions B. The judge model is smaller than the generator, so it cannot properly assess the outputs C. The eval runs each case only once, so more repetitions are needed to average away variance D. Helpfulness is inherently unmeasurable by an LLM and should be dropped from the eval suite
Q10
After tightening a moderation prompt to cut harmful outputs, users complain that benign questions are now refused. Leadership asks what the safety eval should report going forward. What do you recommend?
A. Harmful-compliance rate alone, since harmful content reaching users is the real regulatory exposure B. A single blended accuracy number so stakeholders have one clear metric to track over time C. Nothing new — Constitutional AI training already balances refusals, so app-level evals add little D. Over-refusal on a benign set and harmful slip-through on an adversarial set, as separate metrics
Q11
A production agent approved a transaction it should have flagged. Compliance asks you to reconstruct exactly why. Which logging design would have made this possible?
A. Final assistant responses stored with timestamps and user identifiers for every conversation B. Request/conversation IDs, each tool call's name, arguments, and result, plus provenance tags on facts C. Verbose logging enabled on the model provider's side, with logs requested whenever incidents occur D. Full raw prompts and completions captured for a 1% random sample of production traffic
Q12
An interactive assistant has a 9,000-token stable system prompt (persona, policies, tool definitions) and strict first-token latency targets. Which optimization gives the largest latency and cost win without hurting quality?
A. Move the traffic to the Message Batches API to capture its 50% input and output discount B. Cut the few-shot examples from the system prompt to shrink the tokens processed per call C. Mark the stable prefix with cache_control so subsequent calls read it at ~10% cost and lower latency D. Split the system prompt across several user turns so no single request carries the full load
Q13
An A/B eval of a new routing prompt shows aggregate accuracy up from 82% to 86%, but the refund class — 5% of traffic and the highest-stakes intent — dropped from 91% to 70%. What is the right release decision?
A. Ship v2, since aggregate accuracy is the primary metric and it clearly improved B. Ship v2 and rely on production monitoring to catch refund complaints as they arise C. Revert to v1 permanently, since the experiment shows prompt changes are too risky here D. Hold the release and iterate on v2 until the refund-class regression is resolved
Q14
Finance mandates a 60% cost cut for a synchronous document-classification service currently running on Sonnet. Which sequence best protects quality while pursuing the cut?
A. Benchmark Haiku against Sonnet on the eval set, then downgrade only where Haiku holds the accuracy bar B. Switch all traffic to Haiku immediately and treat support tickets as the quality signal during bake-in C. Move the service onto the Message Batches API, since its 50% discount nearly meets the target alone D. Keep Sonnet but shrink the eval suite, since evaluation runs are a controllable share of the spend
Q15
You are deciding whether a 60KB reference document plus tools and history will fit a model's context budget with headroom. The character-based estimate lands close to the limit. What should you do?
A. Trust the ~4-characters-per-token heuristic, since it is considered accurate at this scale B. Measure with count_tokens and set explicit budgets for prefix, history, and current turn C. Truncate the reference document by half so the estimate is no longer close to the limit D. Switch to the largest-context model available so the estimate no longer constrains the design
Q16
A banking agent told a user their balance covered a transfer; it did not, and the transfer failed. The trace shows a balance tool call from 40 turns earlier whose result was accurate at that time. How do you classify and fix the failure?
A. Hallucination — add grounding instructions so the agent only ever cites retrieved facts B. Prompt failure — the system prompt never warned the agent to treat balances carefully C. Stale data — timestamp tool results and refresh volatile facts on need rather than remembering them D. Model mismatch — a stronger model would have recognized on its own that a re-check was required
Answers
Q1: C. An eval set should be drawn from the real production distribution and deliberately cover typical, ambiguous, and refusal/edge cases — diversity beats similarity. Synthetic-only data (A) misses real-traffic quirks the classifier will actually face. Recency-only sampling (B) under-represents rare intents and edge cases. Hardest-only (D) can't detect regressions on the common traffic that dominates production.
Q2: A. The documented discipline is to compare two prompt versions side-by-side on the same eval set, tracking per-class results so a win in one class doesn't mask a regression in another. (B) is slow, confounded by traffic mix, and puts users in the blast radius. (C) encodes the misconception that eval sets get "biased" like training data — changing the set makes the comparison unmeasurable. (D) is anecdote, not evaluation.
Q3: B. The model is filling gaps from prior knowledge — a grounding failure, and grounding instructions ("Answer from <context> only; if absent, say 'I can't find that'") are documented to dramatically reduce hallucinations. (A) misdiagnoses it as model mismatch: a stronger model just fabricates more fluently without grounding rules. (C) misdiagnoses it as retrieval failure — the facts don't exist in the corpus, so more chunks can't supply them. (D) shortens the fabrication without preventing it.
Q4: D. Forty same-priority rules in a 3,200-token prompt is classic dilution: the model weights no single rule strongly enough. The documented mitigations are moving critical rules to the top and end (primacy + recency) or restructuring/splitting — architectural fixes, tried before a model change. (A) is the band-aid misdiagnosis; the root cause is prompt structure. (B) applies a hallucination fix to an instruction-following failure. (C) is wrong because the 60% failure rate is stable and reproducible — measurement isn't the problem.
Q5: A. Sustained cache_creation traffic with near-zero reads is the documented signature of cache churn: any change to the cached prefix — even one character, like a timestamp or user ID before the breakpoint — invalidates it, so every request pays the ~125% write premium. (B) would show growing input_tokens, not writes on every request. (C) cannot explain writes on messages sent seconds apart — those would hit a still-live cache; TTL expiry produces writes correlated with long gaps, not near-universal writes. (D) describes behavior that doesn't exist — the limit is simply up to 4 breakpoints per request.
Q6: D. The default ephemeral cache expires 5 minutes after last use, and requests arrive ~20 minutes apart — so there are never any reads, and every request pays the ~125% cache-write premium on 40KB, costing more than no caching at all (the fix is the 1-hour TTL tier where available, or accepting uncached calls). (A) has the wrong rate — reads are ~10%, and no reads are occurring anyway. (B) and (C) describe limits and billing rules that don't exist.
Q7: B. Evaluation harnesses are a canonical Message Batches use case: 50% off input and output, results within 24h (fine for nightly), a single batch can mix models for exactly this comparison, custom_id maps results back to test cases, and failures are isolated per request. (A) does the job but forfeits the 50% discount. (C) optimizes first-token UX, which is irrelevant to an offline harness. (D) adds agent-loop cost and nondeterminism to a fixed, predetermined iteration.
Q8: C. Deterministic fields deserve deterministic grading — schema validation and per-field accuracy checks run programmatically — while the required source pointer per field exists precisely to enable audit-grade human spot-checks of a sample. (A) uses a noisy holistic judge for values that can be checked exactly. (B) doesn't scale in cost and adds little over spot-checking when programmatic checks pass. (D) is brittle: formatting or field-order differences fail the match even when every extracted value is correct.
Q9: A. An LLM evaluator is only effective when there are clear evaluation criteria it can apply consistently; without a rubric its scores are noise, and optimizing against noise drives prompt changes in random directions. (B) invents a model-size requirement that isn't the documented condition. (C) is the tempting runner-up — averaging stabilizes the number but the number still measures nothing without criteria. (D) overcorrects: given a rubric, quality dimensions like helpfulness are exactly what judge-based evals are for.
Q10: D. Safety has two failure directions — harmful compliance and over-refusal (HHH: harmless, but don't over-refuse) — and they trade off, so each needs its own dataset (adversarial vs benign-but-edgy) and its own separately tracked metric. (A) ignores the helpfulness half that users are already complaining about. (B) is dangerous because a blended number can improve while one direction quietly regresses. (C) is the misconception that model-level training exempts you from application-level evaluation.
Q11: B. The documented observability bar is audit-friendliness: a regulator should be able to trace any decision back to source data, which requires request/conversation IDs to thread the trace, every tool call's name, arguments, and result, and provenance tags on the facts the agent relied on. (A) captures what the agent said but not why. (C) misplaces responsibility — the API is stateless and the application owns its observability. (D) sampling almost certainly missed the one incident you must reconstruct.
Q12: C. Prompt caching is the documented cost and latency lever for exactly this shape — a large stable prefix reused across interactive calls: reads bill at ~10% of input cost and return faster. (A) fails the constraint outright — batch is async with a 24h SLA and cannot serve interactive traffic. (B) saves tokens but sacrifices the output quality the few-shots provide. (D) reduces nothing — the tokens are still processed — and destabilizes the prefix structure that caching and consistency depend on.
Q13: D. The documented discipline is to track per-class regressions because a change that helps one input class may hurt another — a 21-point drop on the highest-stakes intent is a blocking regression regardless of the aggregate gain. (A) is exactly the aggregate-metric trap. (B) ships a known high-stakes defect and outsources detection to angry customers. (C) overcorrects — the eval did its job; the response is to iterate v2 against the eval set, not abandon prompt improvement.
Q14: A. Measure before you migrate: run the cheaper model against the existing eval set (a mixed-model batch does this at half price) and downgrade only where accuracy holds — the fleet-of-Haikus economics only work when an eval confirms the quality bar. (B) turns customers into your eval harness. (C) fails the stated constraint — the service is synchronous, and batch is async-only. (D) attacks the wrong cost line and removes the regression protection every other option depends on.
Q15: B. The ~4-chars/token figure is explicitly an estimate; when a decision lands near a hard limit, the documented practice is to measure with count_tokens and manage explicit budget categories (stable prefix, history, working memory, current turn). (A) is how context-overflow bugs ship — heuristics are for rough sizing, not boundary calls. (C) destroys reference content to solve a measurement problem. (D) buys cost and latency to avoid a free precise measurement.
Q16: C. This is the documented stale-data hazard: tool results like balances expire, so facts should carry capture timestamps and volatile values should be refreshed on need — refresh-on-need beats remember-forever. (A) fails because the agent did cite a retrieved fact; it was just 40 turns old. (B) is the tempting runner-up, but a prose warning doesn't fix a data-lifecycle defect — the re-read must be a designed behavior. (D) hopes model scale substitutes for a reliability primitive (defensive reads) that has to be engineered in.