Practice — CCAR-P Domain 6: Stakeholder Communication & Lifecycle Management (14%)

13 scenario-based MCQs. Answer key + explanations at the bottom.

---

Q1

You are running the first discovery workshop for an insurance claims assistant. The business sponsor opens with a ranked feature wishlist and asks you to start estimating. Applying structured discovery, which question should you ask first?

A. "Which Claude model do you want the program to standardize on across all teams?" B. "Can you walk us through the feature wishlist again, ranked by the business value the sponsor expects each item to deliver?" C. "What latency targets, request volumes, and data-residency rules must this service operate under?" D. "Which internal teams will consume the assistant's API once it launches?"

Q2

During contract negotiation for an LLM-backed support assistant, procurement insists on writing "99.9% response accuracy" into the SLA. Which counter-proposal best reflects sound practice for LLM-backed services?

A. Accept the term as written — forcing structured output via tool use makes accuracy effectively deterministic B. Offer latency and availability SLAs, and express quality as an SLO measured against a versioned eval set on an agreed review cadence C. Decline any quality commitment whatsoever on the grounds that model outputs are inherently nondeterministic D. Accept the term but insert an evaluator-optimizer loop that reviews every response, so the 99.9% accuracy target becomes guaranteeable in production

Q3

A steering committee of non-technical executives must choose between a predefined pipeline (workflow) and an autonomous agent for document triage, and you get one slide to frame the decision. Which framing is accurate?

A. "The workflow is more predictable and cheaper per document; the agent handles unanticipated cases but trades latency and cost for that flexibility" B. "The agent is strictly more capable than the pipeline, so the workflow only makes sense as a temporary stopgap until the agent is production-ready" C. "The workflow eliminates hallucination risk entirely, while the agent reintroduces that risk through its open-ended reasoning" D. "The two options cost roughly the same to operate at our volume, so the decision reduces to which one can ship faster"

Q4

A team will submit nightly compliance summaries through the Message Batches API at 11 p.m. to capture the 50% discount, and has told the operations VP that results are "guaranteed by 6 a.m." Reviewing the design, what should you flag?

A. Batches cannot mix model tiers, so the projected discount will not apply to their planned Haiku-plus-Sonnet split B. Requests inside a batch cannot benefit from prompt caching, so the cost projection is materially overstated C. Errors in individual requests cause the whole batch to fail, so one bad row would forfeit the entire night's run D. Batch results are promised only within 24 hours, so a 7-hour hard deadline exceeds what the service supports

Q5

Two weeks after launch, the product owner forwards three screenshots of wrong answers and an engineer proposes rewording the system prompt that same afternoon. Which response best establishes a durable feedback loop?

A. Ship the reworded system prompt that same afternoon and monitor complaint volume over the following week to confirm the change helped B. Turn the flagged conversations into eval cases, baseline the current prompt on the full eval set, and A/B the revision before rollout C. Route the affected intents to human agents until complaint volume returns to its pre-launch baseline D. Recommend moving to a larger model, since recurring accuracy complaints usually signal insufficient capability

Q6

You are writing the architecture document a non-ML risk committee must approve before a claims agent touches live customer data. Which content best serves that audience?

A. Data-flow diagrams of where customer data enters and leaves, the human escalation paths, and how decisions trace to source data B. The complete system prompt and few-shot examples, so the committee can inspect exactly what instructions the model receives on every request C. Detailed per-model token pricing tables, since operating cost is ultimately what approval committees are asked to sign off on D. A benchmark appendix comparing the agent's task accuracy against published results from competitor products

Q7

A 60-day pilot of a contract-review assistant is ending and the sponsor asks what evidence would justify promoting it to production. Which readiness criterion is most defensible?

A. The pilot completed its full 60 days without a single service outage or unhandled API error B. Pilot users rate the assistant favorably on the end-of-pilot satisfaction survey and ask to keep using it C. Pre-agreed eval targets are met, and escalation, observability, and iteration-cap safeguards are in place and exercised D. The assistant has been upgraded to the newest model release so production begins on the strongest available version

Q8

The CFO reviewing your cost projections asks why per-request input costs for the RAG assistant are modeled at a fraction of list price. The design caches a large, stable knowledge base in the system prompt. Which explanation is accurate?

A. Cached tokens become free after the first request that writes them, so steady-state requests are billed only for output tokens B. Prompt caching applies a flat 50% discount to both input and output tokens, the same discount mechanism the Batch API uses C. The knowledge base is persisted on Anthropic's servers indefinitely and billed as monthly storage rather than per-request input D. Cache writes cost about 125% of list input price, but reads cost roughly 10%, so steady traffic within the TTL pays a tenth for the cached prefix

Q9

Finance escalates: actual API spend is nearly triple the projection in your architecture document, which assumed a 90% cache-read rate. Logs show each tenant sends requests roughly every 20 minutes. What is the most likely explanation to present?

A. The ephemeral cache expires 5 minutes after last use, so requests spaced ~20 minutes apart re-write the prefix at above-list cost each time B. The request structure exceeded the limit of four cache breakpoints per request, which silently disabled caching across every request C. Output tokens are never cacheable, and the assistant's responses have been running substantially longer and more detailed than the original projection assumed D. The 50% Batch API discount lapsed after the first billing cycle, which effectively doubled the per-request price going forward

Q10

The general counsel must approve one of two designs for a policy Q&A assistant: inserting the entire policy corpus into every request, or retrieving the top-K relevant chunks. She asks for the tradeoff in plain language. Which summary is correct?

A. "Sending everything guarantees grounded answers because the model sees every policy; retrieval risks silently missing relevant documents and only suits teams whose budgets cannot absorb full-context calls" B. "Retrieval is strictly more accurate because smaller contexts eliminate hallucination; sending everything is only justified when the corpus is tiny" C. "Sending everything is simpler but costs more per call and the model weights the middle of long contexts less; retrieval is cheaper and traceable but only as good as what it retrieves" D. "With prompt caching enabled the two designs converge to the same cost, so the decision comes down to engineering preference and time-to-market"

Q11

Anthropic announces that the model behind your production underwriting assistant will be deprecated in 90 days. Which migration plan best manages the lifecycle risk?

A. Switch the model ID on deprecation day without further work, since successor models supersede older ones on every axis B. Freeze the system on the current model and negotiate an extended support window so the architecture never has to change C. Announce a firm cutover date to all stakeholders immediately, since early communication matters most, and update the architecture documents once the swap is complete D. Re-run the eval suite on the successor, A/B against today's baseline, revise cost and latency projections, and schedule cutover with a rollback path

Q12

A compliance requirement states that a customer-facing assistant must be transparent about its limitations. Which design decision best fulfills the obligation without degrading usefulness?

A. Append a disclosure of the underlying model version and parameter count to the footer of every response the assistant sends B. Instruct it to say when an answer is not in the provided context rather than answering from prior knowledge, and to escalate to a human at low confidence C. Publish the complete system prompt on the company's public website so any customer can read the assistant's operating instructions D. Have the assistant preface every single answer with a standing uncertainty caveat so that users never place excess confidence in any response it produces

Q13

A customer contract for a synchronous, interactive Claude-backed API needs a latency commitment, but response length varies widely with query complexity. Which SLA formulation is most defensible?

A. A fixed 3-second maximum for every request, since generation latency is stable once the model tier has been chosen B. No latency commitment at all, since token-by-token generation time cannot be meaningfully bounded in advance C. Percentile targets (e.g., p95) on time-to-first-token with streaming for the interactive UX, plus a separate availability commitment D. A commitment on mean end-to-end latency only, since averaging across many requests smooths out the long-tail variance that individual queries introduce

---

Answers

Q1: C. Structured discovery means surfacing the constraints that shape the architecture early — latency, volume, and data-residency determine workflow-vs-agent choice, model selection, and compliance posture before any feature is estimable. Re-ranking features (B) is the tempting runner-up, but a wishlist without constraints cannot be translated into an architecture. Model choice (A) is an output of discovery, not an input, and consumer teams (D) matter later for integration, not for shaping the core design.

Q2: B. For LLM-backed services you commit contractually to what you control — latency and availability targets — and express quality as an SLO measured on a versioned eval set with a review cadence, which gives procurement something enforceable and measurable under nondeterminism. Refusing any quality commitment (C) is the runner-up but abdicates accountability and usually kills the deal. Tool use (A) forces structure, not correctness; an evaluator-optimizer loop (D) improves quality but cannot make a probabilistic system hit a guaranteed accuracy number.

Q3: A. This is the source tradeoff verbatim: workflows follow predefined code paths and are predictable and cheap, while agentic systems "trade latency and cost for better task performance." (B) is the runner-up misconception — agents are not strictly better; the guidance is to use the simplest pattern that works. (C) overclaims, since workflows still contain LLM calls that can hallucinate, and (D) is factually wrong on cost parity.

Q4: D. The Batches API is asynchronous with results promised within 24 hours (usually faster) — committing to a 7-hour hard deadline builds an SLA the underlying service does not support. (B) is the runner-up but false: requests within a batch do benefit from prompt caching. (A) is false because a batch can mix models, and (C) is false because failures are per-request while the batch as a whole still completes.

Q5: B. The discipline is: never tweak a prompt without an eval set — convert the real failures into eval cases, baseline, then A/B the change, because a "better" change for one input class may regress another. Shipping and watching complaints (A) is the runner-up but loses on regression protection: complaint volume is a lagging, noisy signal that cannot detect what the reword broke elsewhere. (C) treats the symptom without generating learning, and (D) reaches for capability when the evidence has not localized the cause.

Q6: A. A non-ML risk committee approves controls, not internals: where data flows, when humans take over, and whether any decision can be traced back to source data — the audit standard is that a regulator can reconstruct any decision. The raw system prompt (B) is the runner-up trap: it feels like transparency but is implementation detail the audience cannot evaluate, and it goes stale with every revision. Pricing tables (C) and competitor benchmarks (D) answer questions this committee is not chartered to ask.

Q7: C. Production readiness is demonstrated quality plus operational safeguards: hitting pre-agreed eval targets shows the system works, and exercised escalation paths, observability, and iteration caps show it fails safely. Uptime alone (A) is the runner-up but measures availability, not answer quality — an assistant can be confidently wrong for 60 straight days. Satisfaction surveys (B) are subjective and unanchored, and swapping to a newer model at cutover (D) invalidates the pilot's evidence rather than strengthening it.

Q8: D. Prompt caching prices are asymmetric: the first write costs ~125% of list input, subsequent reads cost ~10%, so it pays off after roughly two reuses and steady traffic inside the TTL window pays about a tenth for the cached knowledge base. (A) is the runner-up but overstates the benefit — reads are discounted, not free, and the write premium is real. (B) confuses caching with the Batch API's 50% discount, and (C) invents persistent server-side storage; the cache is ephemeral with a TTL.

Q9: A. The default ephemeral cache lives 5 minutes after last use; at ~20-minute request spacing every request misses, re-writing the prefix at ~125% of list price instead of reading at ~10% — exactly the gap between projection and actuals. (C) is the runner-up: output tokens are indeed never cached, but the logs point at request spacing, not response length. The four-breakpoint limit (B) is real but exceeding it is not the failure mode shown here, and (D) fabricates a discount expiry that does not exist.

Q10: C. This is the documented tradeoff table: dumping the corpus is simple but expensive, slower, and suffers "lost in the middle" weighting; classic RAG is cheap, fast, and traceable, with retrieval quality as the bottleneck. (A) and (B) are mirror-image overclaims — full context does not guarantee correctness, and small context does not eliminate hallucination. (D) is the runner-up for a caching-savvy audience but wrong: caching discounts the cached prefix, it does not equalize two designs with different token footprints and TTL exposure.

Q11: D. A model change is an architecture change: re-run the eval suite on the successor, A/B against the current baseline to catch regressions, revise cost and latency projections (a different model ID also invalidates the prompt cache), and cut over with rollback available. (C) is the runner-up — communicating early is right, but announcing a firm date before validation commits stakeholders to an unverified plan. (A) assumes newer means regression-free for your workload, and (B) fights a deprecation you do not control.

Q12: B. Honest limitation-handling lives in behavior: refusing to answer beyond the provided context and escalating to a human at low confidence — "I don't know" is a valid agent action and better than hallucinating. (D) is the runner-up but over-refuses: blanket uncertainty caveats degrade helpfulness and train users to ignore the warnings. Model-version footers (A) and publishing the system prompt (C) are disclosure theater — they inform no customer decision and fulfill no behavioral obligation.

Q13: C. Under nondeterministic, length-dependent generation, the defensible commitments are percentile-based latency targets on what you can control — time-to-first-token, delivered via streaming for interactive UX — paired with an availability commitment. (D) is the runner-up but means hide exactly the tail behavior customers experience and contracts get disputed over. A fixed universal maximum (A) is indefensible when output length varies, and offering no commitment (B) abdicates the architect's job of defining SLAs under nondeterminism.