Practice — CCDV-F Domain 4: Eval, Testing, and Debugging (2.6%)
8 scenario-based MCQs. Answer key + explanations at the bottom.
Q1
Your invoice-extraction service intermittently hands unparseable JSON to downstream consumers. You pull traces for the failing requests: every one ends with stop_reason: "max_tokens" and the JSON body is cut off mid-array, while successful requests end with stop_reason: "end_turn". What is the correct diagnosis?
A. Model fault — Claude is emitting malformed JSON, so the prompt needs stronger formatting instructions
B. Model fault — sampling temperature is too high, so the output schema drifts on longer invoices
C. Integration fault — the configured max_tokens cap truncates long outputs before the JSON closes
D. Integration fault — the prompt cache expired mid-generation, dropping the tail of the response
Q2
A team generates structured JSON via a forced tool call and validates every result against a Pydantic schema. About 2% of generations fail validation. Which recovery strategy best matches the recommended pattern?
A. Re-prompt with the validator's error message included, cap retries at about 3, then route persistent failures to human review
B. Re-send the identical request until a generation passes, since low-frequency validation failures are random sampling noise
C. Escalate every validation failure to a larger model immediately, keeping the original prompt unchanged
D. Loosen the schema by removing fields from required until the observed failure rate reaches zero
Q3
You're debugging a home-grown tool loop. In a failing trace, Claude returns a tool_use block for get_weather, your code runs the tool, and the next request appends a user message whose content is the plain string "72F and sunny". From then on, Claude keeps re-requesting the same tool call as if the tool never ran. What is the fix?
A. Set tool_choice to none on the follow-up request so Claude stops re-requesting the tool
B. Move the tool output into the system prompt, where Claude weights it more heavily than user turns
C. Resend the assistant's tool_use block with an is_error: false field attached to mark it complete
D. Return the output as a tool_result content block whose tool_use_id matches the tool_use block's id
Q4
A chat product built on the Messages API occasionally renders a blank assistant message, and users report the bot "goes silent" on questions that need a lookup. Traces of the silent turns show content containing only a tool_use block and stop_reason: "tool_use"; the app rendered nothing and waited for the user. How should you classify and fix this?
A. Model fault — Claude failed to produce any text; add an instruction that every reply must include prose for the user
B. Integration fault — the loop treats these turns as final; it must branch on stop_reason, run the tool, and call back
C. Model fault — the tool description is too vague, causing spurious calls that should be suppressed with tool_choice: "none"
D. Integration fault — max_tokens is set too low to fit both prose and a tool call, so the text block gets dropped
Q5
A team runs a nightly regression suite of 200 prompts and tracks pass rates. Over two weeks the pass rate slid from 96% to 89% with no changes to prompts, harness code, or eval data. The harness calls the API with a model alias rather than a full versioned ID. What should the team change first so runs are comparable?
A. Pin an exact model ID in the harness so the underlying model cannot shift when the alias moves
B. Set temperature to 0 in every eval call to eliminate sampling variance between runs
C. Add prompt caching so every nightly run scores against the same cached prefix
D. Rerun each prompt five times and take a majority vote to smooth out per-run noise
Q6
Latency and cost for a support assistant crept up after a release. You audit usage across a day of traffic: nearly every request reports large cache_creation_input_tokens and cache_read_input_tokens: 0, even for requests seconds apart in the same conversation. What is the most likely defect?
A. The 5-minute TTL is expiring between requests, so each call must rewrite the cache from scratch B. The team set more than one cache breakpoint, which forces a full cache rewrite on every request C. Requests are exceeding the context window, which silently disables caching for the overflow D. The release put a volatile value, such as a per-request timestamp, into the cached prefix, invalidating it on every call
Q7
In a 40-turn support session, Claude quotes a customer's account balance incorrectly at turn 38. The trace shows get_balance was called once at turn 6, the customer made a payment mid-session through another channel, and no later balance lookup appears — the quoted figure exactly matches the turn-6 result. What is the right conclusion?
A. Model fault — Claude hallucinated a plausible figure and needs a stronger grounding instruction in the system prompt B. Integration fault — a volatile fact went stale in context; the agent should refresh volatile data on need rather than reuse old tool results C. Model fault — "lost in the middle" made Claude misread the turn-6 result, so the history window should be shortened D. Integration fault — the tool result should have been cached with a 1-hour TTL so the balance stayed consistent
Q8
During a database outage, an internal agent kept confidently reporting that records were updated. Traces show its update_record tool wrapper catches all exceptions and returns an empty string as the tool_result content. Which change best fixes this failure mode?
A. Wrap the tool in client-side retries and only surface a result once the database has recovered
B. Add a system-prompt rule: "If a tool returns an empty result, tell the user the operation failed"
C. Return tool_result with is_error: true and a clear message, so Claude sees the failure and can retry or escalate
D. Abort the entire agent loop with an unhandled exception whenever any tool raises
Answers
Q1: C. stop_reason: "max_tokens" means the completion hit the hard output cap — the JSON isn't malformed, it's truncated, which is an integration-side configuration fault. (A) is the tempting runner-up but loses because the trace signal appears on every failure and correlates with truncation mid-array, not generation quality. (B) temperature affects variability, not where output stops. (D) prompt caching applies to input tokens and cost; it never alters generated output.
Q2: A. The documented validation loop is: generate, validate, re-prompt with the validator's error message, cap retries (~3), and route to human review if still failing. (B) loses because a blind resend gives the model no signal about what was wrong and is unbounded. (C) escalates cost without a diagnosis — the feedback loop should be tried first. (D) makes failures disappear by destroying the contract downstream consumers depend on.
Q3: D. The tool-use protocol requires the follow-up user turn to carry tool_result blocks whose tool_use_id matches the tool_use block — plain text is not recognized as the answer to the pending call, so Claude re-requests. (A) suppresses the symptom but Claude still never receives the result tied to its call. (B) misplaces per-turn data in the system prompt and still leaves the pending call unanswered. (C) is malformed — is_error belongs on tool_result, not tool_use.
Q4: B. stop_reason: "tool_use" means Claude is asking the application to run a tool and call back; treating that turn as a final answer is a loop bug, so the fix is to branch on stop_reason. (A) misreads the protocol — a tool-only turn is correct behavior mid-loop. (C) loses because the calls are appropriate: the questions genuinely need a lookup. (D) doesn't fit the trace, which would show stop_reason: "max_tokens" if the cap were the problem.
Q5: A. A sustained multi-week slide with nothing else changing points at the model itself shifting under the alias; pinning an exact model ID removes the confound so regression results are comparable across runs. (B) is the runner-up: temperature 0 reduces run-to-run sampling variance but cannot explain or prevent a directional drift caused by the alias re-pointing. (C) caching changes cost and latency, not outputs. (D) smooths noise while leaving the underlying confound in place.
Q6: D. Sustained cache_creation traffic with zero reads means the cacheable prefix changes on every request — the classic cause is volatile data (timestamps, request IDs) placed before the breakpoint, which invalidates the cache each call; volatile data belongs after the breakpoint. (A) is the runner-up but is ruled out by requests seconds apart still missing — TTL can't have expired. (B) up to 4 breakpoints are supported and normal. (C) context overflow raises errors; it doesn't silently disable caching.
Q7: B. The quoted figure traces exactly to the turn-6 tool result, so the model faithfully used what its context contained — the defect is a design that let a volatile fact go stale instead of refreshing on need. (A) loses because a value that matches an earlier tool result is stale data, not hallucination. (C) Claude read the value correctly; shortening history doesn't make old data current. (D) prompt caching stores tokens for cost savings — it does nothing to keep facts fresh.
Q8: C. Silent failure is a documented tool-use pitfall: return is_error: true with a clear message so Claude can see the failure and retry or escalate instead of narrating success. (A) is the runner-up but stalls the turn indefinitely during an outage and still hides the fault from the model. (B) is fragile prompting around a protocol-level signal — an empty string can also be a legitimate success value. (D) removes Claude's ability to recover gracefully or explain the failure to the user.