Practice — CCDV-F Domain 5: Model Selection and Optimization (16.8%)
13 scenario-based MCQs. Answer key + explanations at the bottom.
Q1
Your team is scoping a new document-triage feature and has not yet validated that an LLM can do the task at all. The eventual production budget strongly favors a cheap model. According to the recommended model-selection workflow, what should you do first?
A. Start on Haiku so that early cost data reflects the production budget from day one B. Run the same prompt on all three model tiers in parallel and average the results C. Prototype on a strong model to prove the task is feasible, then downshift and measure D. Start on Sonnet and never change models, since switching models invalidates prompt caches
Q2
You need a quick feasibility check: will a ~600,000-character contract corpus fit into a single request on a model with a 200K-token context window? Using the standard rough conversion, what do you conclude?
A. ~600K tokens — it exceeds the window by roughly 3x, so it must be chunked B. ~150K tokens — it fits, and you should confirm with the count_tokens helper before shipping C. ~450K tokens — applying the standard 0.75 conversion factor, it exceeds the window by more than 2x, so it must be chunked D. ~75K tokens — it fits easily, with more than half of the window left to spare
Q3
A team sets temperature to 0 on a regression suite that asserts byte-identical model outputs against golden files. The suite still flakes: outputs occasionally differ across runs even though the requests are identical. What is the best explanation and fix?
A. top_p must also be zeroed; setting both parameters together guarantees identical outputs B. The model alias must have moved overnight; pinning the exact model ID makes outputs byte-identical C. This is an API defect; add retry logic that re-runs the request until the output matches the golden file D. temperature 0 reduces randomness but never guarantees identical outputs; assert on semantic or structural properties instead
Q4
You run two workloads: (1) a customer-facing streaming chat where time-to-first-token drives satisfaction scores, and (2) a nightly job that reviews pull requests for subtle correctness bugs. Where should extended thinking be enabled?
A. On the nightly review job only — it helps hard reasoning, while its latency and output-token cost hurt the streaming chat B. On the streaming chat only — thinking lets the model plan the whole answer in advance, which speeds up delivery of the first visible token C. On both workloads — thinking is billed as input tokens, so it is effectively free once the prompt is cached D. On neither workload — thinking only improves math problems, and neither workload is mathematical
Q5
An agent loop with extended thinking enabled works on single-turn requests but degrades sharply during multi-turn tool use. You find that the history-builder "sanitizes" each assistant turn, keeping only text and tool_use blocks before replaying the conversation. What is the correct fix?
A. Raise the thinking budget so the model has enough room to re-derive its reasoning from scratch at the start of every turn B. Replace each thinking block with a short plain-text summary to save on replayed tokens C. Pass prior thinking blocks back unchanged, including their signatures, when continuing tool use D. Drop only the redacted_thinking blocks, since their filtered content is unreadable anyway
Q6
A fleet of narrowly scoped subagents performs simple lookups inside a larger agent system. Traces show long preambles and many small tool calls per subagent turn, and token spend is dominated by these subagents. Which single request-level change most directly addresses this?
A. Set output_config effort to low on the subagent requests B. Reduce max_tokens so that subagent responses are forced shorter C. Set temperature to 0 so the subagents stop producing exploratory text D. Remove the subagents' cache breakpoints to shrink the prompt prefix
Q7
Your interactive coding product runs on an Opus-tier model, and users complain that long generations stream too slowly. An engineer proposes enabling fast mode. Which statement about fast mode is accurate?
A. It routes the request to a smaller distilled model, trading answer quality for speed B. It serves the same model at substantially higher output tokens per second, at premium per-token pricing C. It is a discount tier like batch processing: slower scheduling in exchange for 50% lower cost D. It changes only the network transport layer, so throughput, quality, and pricing stay the same
Q8
An extraction prompt returns inconsistently formatted output, so a developer adds exactly one worked example. Format compliance improves, but the model now copies that example's quirks — even echoing its field values on unrelated inputs. What is the best next step?
A. Remove the example entirely and expand the prose instructions to describe the format B. Keep the single example but move it from the system prompt into the user turn C. Add fifteen or more near-identical examples to anchor the format as strongly as possible D. Provide three to five diverse examples covering typical, ambiguous, and edge-case inputs
Q9
A support platform handles 50,000 requests per day: roughly 90% are simple FAQ-style questions and 10% require multi-step account analysis. Leadership wants cost per ticket minimized without degrading the hard cases. Which model strategy fits best?
A. Run Sonnet on all traffic to maintain a single, predictable quality bar B. Classify with Haiku, answer simple tickets on Haiku, and escalate complex tickets to Sonnet C. Run Opus on all traffic so that the hardest 10% of tickets is never under-served D. Run Haiku on all traffic and re-run any user-flagged failures on Opus afterward, so hard cases eventually reach a stronger model
Q10
Overnight, with no deploy, a production feature's outputs change tone and format, and one prompt regression appears. The request code specifies the model with a moving alias that always tracks the newest release. What is the best remediation?
A. Set temperature to 0 so the new model release behaves like the previous one B. Add stop_sequences to the request to constrain the new model's format drift C. Pin an exact model ID, and adopt new releases deliberately behind your eval suite D. Enable prompt caching so that cached responses preserve the previous behavior
Q11
You enable prompt caching on an endpoint that serves several requests per minute, all sharing a large system prompt. A week later the bill is slightly higher than before. The usage blocks show cache_creation_input_tokens roughly equal to the prefix size on nearly every request, and cache_read_input_tokens near zero. What is the most likely cause?
A. A volatile value (such as a timestamp or user ID) sits inside the cached prefix, so every request re-writes the cache and never reads it B. Requests are spaced further apart than the cache TTL, so each entry expires before the next request arrives and every request becomes a fresh cache write C. The ~125% write premium always outweighs the ~10% read price, so caching raises costs on any workload D. Prompt caching is not enabled for this account, so the usage fields are reporting placeholder values
Q12
A chat endpoint sends about 200 requests per hour, each carrying the same 20,000-token cached system prefix plus ~500 variable tokens, and requests arrive well inside the cache TTL. After the first request in each window, roughly what does each request pay for the 20K-token prefix, relative to the uncached input price?
A. ~0% — once the prefix has been written to the cache, subsequent in-TTL requests read it back for free B. ~50% — cached tokens receive the same discount as the Batch API C. ~125% — every request pays the cache-write premium on the prefix D. ~10% — cache reads are billed at about a tenth of the normal input price
Q13
An assistant's prompt contains: a fixed persona, a large reference document updated weekly, a growing conversation history, and a per-request block with the current timestamp and user context. How should cache breakpoints be arranged?
A. Place a single breakpoint on the per-request block, since caching the last block covers everything before it B. Move the timestamp to the top of the system prompt for freshness and set a single breakpoint at the very end C. Breakpoints after the persona, the reference document, and the latest history turn, with the per-request block after the final breakpoint D. Add a breakpoint after every content block so that any prefix length can be served from the cache
Answers
Q1: C. The recommended workflow is to prototype on a strong model to prove the task is possible, then downshift and measure — starting cheap leaves you unable to tell whether failures mean "task infeasible" or "model too weak." (A) makes that ambiguity permanent. (B) burns budget without isolating feasibility. (D) misapplies a real fact: switching models does invalidate a prompt cache, but that is a per-conversation cost concern, not a reason to freeze model choice.
Q2: B. The rough conversion is 1 token ≈ 4 characters, so 600,000 characters ≈ 150K tokens — inside a 200K window, but close enough that you should verify with the count_tokens helper. (A) treats characters as tokens. (C) misapplies the 0.75 factor, which is words per token, not tokens per character. (D) assumes ~8 characters per token, double the standard estimate.
Q3: D. Lowering temperature makes sampling more deterministic, but it has never guaranteed bit-identical outputs — robust regression suites assert on structure or semantics (e.g., parse against a schema) rather than exact strings. (B) is the tempting runner-up: pinning is genuine best practice, but it prevents behavior shifting across releases, not run-to-run sampling variance on the same model. (A) doesn't create a byte-identical guarantee, and top_p is normally left at its default. (C) mislabels expected behavior as a defect.
Q4: A. Extended thinking is for hard multi-step reasoning like correctness-critical code review; it is explicitly discouraged for streaming UIs where first-token latency matters, since the reasoning trace is produced before the visible answer and billed as output tokens. (B) inverts reality — thinking delays first tokens. (C) encodes a real confusion: thinking tokens count toward output cost, not input, so caching does not make them cheap. (D) understates thinking's use cases (planning, tool selection, code, logic — not just math).
Q5: C. When tool use is involved, prior thinking blocks must be included in the next message with their signature fields intact, otherwise Claude loses its reasoning trace — stripping them is exactly the bug shown. (D) is the trap: redacted_thinking blocks must also be passed back unchanged even though they are unreadable. (A) increases cost without restoring the lost trace. (B) alters the block, so the signature no longer accompanies valid content.
Q6: A. The effort level is the per-request dial for reasoning depth and overall token spend: lower effort yields fewer, more consolidated tool calls, less preamble, and terser confirmations — precisely the observed waste, and low effort is the recommended setting for scoped subagents. (B) truncates output mid-flight (risking broken tool calls) instead of making the model plan tersely. (C) changes sampling randomness, not verbosity or tool-call count. (D) would raise cost — breakpoints are what make the repeated prefix cheap.
Q7: B. Fast mode serves the same model at up to ~2.5x higher output tokens per second at premium per-token pricing, with its own rate limit — quality is unchanged because the model is unchanged. (A) is the classic misconception that speed implies a distilled model. (C) confuses fast mode with the Batch API's 50% discount; they sit at opposite ends of the latency/cost trade, and fast mode is not available with batch. (D) is wrong because pricing rises and throughput comes from serving, not transport.
Q8: D. A single worked example is a documented anti-pattern ("zero-shot+one") — the model overfits to it, exactly as observed. The fix is 3–5 diverse examples covering typical, ambiguous, and edge cases, showing the exact output format. (C) loses twice: beyond ~10 examples returns diminish while context cost grows, and near-identical examples teach less than diverse ones. (A) reverts to the original inconsistent-format problem. (B) changes placement, which affects stability vs per-request variation, not overfitting.
Q9: B. Route-and-tier: Haiku is the cost-aligned model for high-volume routing and simple answers, and escalating only the 10% of complex tickets to Sonnet protects quality where it matters. (A) is the plausible runner-up but pays Sonnet rates on the 90% of traffic Haiku handles fine, so it loses on cost. (C) maximizes cost for no quality gain on simple tickets. (D) makes customers the quality gate — hard-case failures ship before anyone detects them.
Q10: C. "Always pin" is the rule: a moving alias means a new release can silently change behavior — and releases can also remove or reject previously valid parameters, breaking requests outright. The fix is an exact pinned model ID plus deliberate, eval-gated adoption of new releases. (A) and (B) are band-aids that do not restore the prior model's behavior. (D) misunderstands caching: it caches prefix processing for cost and latency, not responses or model behavior.
Q11: A. Sustained cache_creation with near-zero cache_read is the signature of prefix churn: a volatile value inside the cacheable prefix invalidates the cache on every request, so each one pays the ~125% write premium and never earns the ~10% reads — which is why the bill went up. (B) is the plausible alternative diagnosis, but it is ruled out by the stated traffic: several requests per minute is well inside the 5-minute TTL. (C) is false — caching pays off after ~2 reuses of a stable prefix. (D) is contradicted by the nonzero cache_creation counts being reported.
Q12: D. Cache reads bill at roughly 10% of normal input price, so after the initial write, every in-TTL request pays ~10% for the 20K-token prefix — at 200 requests/hour the traffic itself keeps the entry warm. (A) overstates it: reads are cheap, not free. (B) confuses prompt caching with the Batch API's separate 50% discount. (C) misapplies the ~125% premium, which is charged on writes only; this workload is almost entirely reads.
Q13: C. Breakpoints belong at stability boundaries — persona (never changes), reference document (weekly), latest history turn (per turn) — with volatile per-request data placed after the final breakpoint, within the 4-breakpoint limit. (A) folds the volatile block into the cached unit, so the prefix differs on every request and is never re-read. (B) is worse: the fastest-changing value sits first, invalidating the entire prefix each request. (D) exceeds the 4-breakpoint limit once history grows and adds nothing a boundary breakpoint doesn't already provide.