Vault / wiki/301/practice/ccao/domain-7-troubleshooting-and-optimization.md
updated 2026-07-16Practice — CCAO-F Domain 7: Troubleshooting and Optimization (10%)
9 scenario-based MCQs. Answer key + explanations at the bottom.
Q1
An operations manager pastes a 90-page vendor contract into a claude.ai conversation and works through it with Claude over an afternoon. After about thirty follow-up exchanges, Claude starts contradicting its earlier answers and overlooking clauses it summarized correctly at the start. Which approach best restores output quality?
A. Switch the conversation to the most capable model, since it can track more detail across long exchanges B. Ask Claude to re-read the full contract carefully before giving its next answer C. Have Claude summarize the key findings and open items, then continue in a fresh conversation seeded with that summary D. Break the contract into thirty separate conversations, one clause per thread, and merge the results manually
Q2
A marketing associate's product-announcement prompt keeps producing generic, off-target copy. A teammate suggests switching to the most capable model tier. According to Anthropic's canonical order for fixing an underperforming prompt, what should the associate try first?
A. Add two or three examples of past announcements that hit the right tone B. Restate the task directly, with explicit constraints and the expected output format C. Move to the most capable model and re-run the identical prompt there D. Assign Claude the role of an expert product copywriter in the instructions
Q3
A consultant asks Claude to draft an industry overview and later discovers that several confidently cited market-share statistics were fabricated. Which change to the workflow best prevents this failure from recurring?
A. Provide the source reports in the conversation and instruct Claude to answer only from the supplied material B. Ask the same question in a brand-new conversation so that stale context cannot interfere with the numbers C. Add formatting instructions so every statistic appears in a table that is easier to check by hand afterward D. Rephrase the request with stronger emphasis on accuracy so Claude takes more care with the figures
Q4
At the start of every new conversation, an education team re-pastes the same 15 pages of curriculum standards and tone guidelines before asking Claude to draft lesson plans. The drafts are fine, but setup is tedious and teammates sometimes paste outdated versions of the guidelines. Which context strategy best fits this situation?
A. Keep one long-running shared conversation alive so the guidelines never need to be pasted again B. Ask Claude to condense the guidelines once and paste that shorter summary at the start of each chat C. Paste the guidelines at the end of each prompt rather than the beginning so they carry more weight D. Persist the guidelines as knowledge files and instructions in a shared Project and draft lessons there
Q5
You instruct Claude at the start of a long working session to "always flag any figure that lacks a source." Early answers comply, but after you paste several lengthy documents, Claude stops flagging unsourced figures. What is the most likely explanation?
A. Claude's knowledge cutoff prevents it from recognizing figures from the newer documents B. The instruction has been diluted by the growing context and needs restating near the latest request C. The conflicting data inside the pasted documents has caused Claude to start hallucinating D. Instructions in claude.ai apply only for a few turns by design, so the rule quietly expired
Q6
A product manager asks Claude about a competitor's product launched last month. The answer is vague, so she rewrites the prompt four times — adding a role, examples, and clearer structure — with no improvement. What best explains the situation?
A. The model tier is too low, because the most capable models are trained on more recent information B. The prompt is still not clear and direct enough, and further rewording will eventually get there C. Accumulated context from her earlier rewrites is interfering, and a fresh conversation would clear it D. The launch likely postdates Claude's knowledge cutoff, so she needs to supply sources or use research
Q7
A sales team's Project contains 60 uploaded files — every battle card, call transcript, and pricing sheet from the past two years. Lately, every conversation in the Project produces weaker, less relevant answers, even on the very first message. Which fix should the team try first?
A. Curate the Project knowledge down to the current, relevant documents and remove the rest B. Tell teammates to open each conversation with a longer, more detailed first prompt C. Switch the Project's conversations to a faster model tier to offset the degradation D. Start a new conversation inside the Project whenever the answers begin to slip
Q8
An analyst needs Claude to reason through a multi-quarter scenario model with interdependent assumptions. The prompt already states the task clearly, supplies all the data, includes worked examples, and specifies the output format — yet the fast-tier model's analysis stays shallow and misses interactions between assumptions. What is the best next move?
A. Add more worked examples to the prompt until the reasoning depth improves B. Split the request across several fresh conversations to reduce the load C. Move the task to a more capable model tier suited to deep multi-step reasoning D. Re-paste the data immediately before the question so it sits closer in context
Q9
Claude's first draft of a customer newsletter misses the mark: it targets the wrong audience and reads far too formally. Which next step best reflects the recommended iteration loop?
A. Identify what went wrong — audience and tone — and revise the request to state those requirements B. Regenerate the response several times and keep whichever draft happens to come out best C. Start over in a new conversation with the identical prompt to get a cleaner attempt D. Reply "try again, but better and less formal" and let Claude infer what needs to change
Answers
Q1: C. The symptoms — contradictions and forgotten details deep into a long thread — point to context-window exhaustion, and the standard remedy is summarize-then-restart: compress the durable findings and seed a fresh conversation. Switching models (A) is the classic wrong-layer distractor; a more capable model does not undo an exhausted context. Asking Claude to "re-read carefully" (B) changes nothing about what fits in working memory, and thirty single-clause threads (D) destroy the cross-clause context the task needs.
Q2: B. Anthropic's canonical order says to walk the prompt levers top to bottom before reaching for a bigger model, and lever one is "be clear and direct" — state the task, constraints, and expected format. Examples (A) and a role (D) are real levers but come later in the order, so they lose on sequence, not validity. Jumping to a stronger model (C) skips the cheapest, highest-yield fixes and often masks a describable problem.
Q3: A. Fabricated statistics mean Claude was asked to recall rather than retrieve — hallucination is next-token prediction operating without grounding — so the fix is to supply source material and constrain answers to it. A fresh conversation (B) misdiagnoses this as a context problem; the fabrication happens on the first message too. A table (C) aids later verification but does nothing to prevent invention, and emphasizing accuracy (D) is a steerability misconception — instructions cannot conjure knowledge the model lacks.
Q4: D. Recurring, durable reference material is the textbook case for the persist strategy: a Project holds the instructions and knowledge files once, gives every teammate the current version, and ends the re-pasting ritual. One eternal conversation (A) degrades as the context fills. A pasted summary (B) still requires manual setup, loses detail, and leaves the version-drift problem unsolved; moving the paste to the end (C) addresses none of the actual pain.
Q5: B. This is the steerability × working-memory collision: as pasted documents pile up, an early instruction gets diluted and loses its grip, and the fix is to restate the constraint near the most recent request and keep the context lean. The knowledge cutoff (A) governs training data, not documents you supply. Conflicting data (C) would produce wrong answers, not a silently dropped rule, and instructions have no built-in turn-expiry (D) — that mechanism is invented.
Q6: D. No amount of prompt iteration can add knowledge the model does not have: events after the knowledge cutoff must be supplied in the conversation or fetched via a search/research capability. A more capable tier (A) is the documented misconception here — capability is not recency. More rewording (B) misreads a knowledge limitation as a description problem, and a fresh conversation (C) restarts with the same missing knowledge.
Q7: A. Project knowledge counts against the context window of every conversation, so 60 files crowd out the user's actual request from the very first message — the guidance is to curate, not dump. Longer opening prompts (B) add even more competing content. A faster model (C) changes speed, not the crowding, and restarting conversations (D) cannot help because the overloaded knowledge loads again each time.
Q8: C. The analyst has already exhausted the prompt levers — clarity, context, examples, format — so the remaining diagnosis is wrong model: deep multi-step reasoning over interdependent assumptions is exactly what the most capable tier is for. More examples (A) improve tone, format, and edge cases but cannot raise a model's reasoning ceiling. Splitting the task (B) severs the interdependencies the analysis needs, and re-pasting data (D) fixes a dilution problem this prompt does not have.
Q9: A. The Description–Discernment loop treats every prompt as a hypothesis: evaluate the output, diagnose the specific gap (wrong audience, wrong tone), and re-describe with those requirements made explicit. Repeated regeneration (B) is gambling, not iterating — nothing about the request improves. Rerunning the identical prompt in a new thread (C) reproduces the same failure, and "try again, but better" (D) skips the discernment step, leaving Claude to guess what "better" means.