Project 16 — Capstone: Customer Support Resolution Agent

Enforces: everything — tool design, structured output, routing/escalation, reliability, evals, and caching (the full course) Surface: code (Python, Anthropic SDK) · Time: ~3–4 hrs · Difficulty: 🔴 capstone

Why this project

This is the capstone. It mirrors the CCA-F "Customer Support Resolution Agent" scenario (cca-scenarios · Scenario 1) — the canonical production agent the exam tests you on. You'll pull together every thread of the course: tools with a confirmation gate, forced-tool structured output, routing and escalation, graceful error handling, an eval set to measure it, and prompt caching to make it affordable. If you can build this, you can build a real agent.

What you'll build

An end-to-end customer support agent built on the Anthropic Messages API (claude-opus-4-8 for the agent, claude-haiku-4-5 for the cheap classifier). It resolves order issues over a multi-turn loop: it looks up orders (read-only), issues refunds only after explicit confirmation, escalates to a human when out of policy or low-confidence, returns a structured verdict per turn, recovers from tool errors, and ships with an eval set and prompt caching on the stable system prompt.

Steps

Define the tools — Three tools, designed by reversibility (see tool-design-principles). lookup_order is read-only; issue_refund is destructive and gated; escalate_to_human is always available.

TOOLS = [
    {
        "name": "lookup_order",
        "description": "Look up an order by ID. Read-only. Call this before any refund decision; never trust stale order data from earlier in the conversation.",
        "input_schema": {
            "type": "object",
            "properties": {"order_id": {"type": "string"}},
            "required": ["order_id"],
        },
    },
    {
        "name": "issue_refund",
        "description": "Issue a refund for an order. DESTRUCTIVE. Only call after the user has explicitly confirmed the amount. Requires confirmed=true.",
        "input_schema": {
            "type": "object",
            "properties": {
                "order_id": {"type": "string"},
                "amount_cents": {"type": "integer"},
                "confirmed": {"type": "boolean", "description": "Must be true; set only after the user confirms the exact amount."},
            },
            "required": ["order_id", "amount_cents", "confirmed"],
            "additionalProperties": False,
        },
    },
    {
        "name": "escalate_to_human",
        "description": "Hand off to a human agent. Use when the request is out of policy, the user is dissatisfied, or you are not confident you can resolve it safely.",
        "input_schema": {
            "type": "object",
            "properties": {"reason": {"type": "string"}, "summary": {"type": "string"}},
            "required": ["reason", "summary"],
        },
    },
]

Gate the destructive action — In your tool executor, refuse to process a refund unless the model passed confirmed=true AND your host code has seen the user confirm the exact amount. This is the confirmation gate — the model asking isn't enough; the host enforces it. Return an error tool_result (is_error=True) if the gate isn't satisfied so the model self-corrects.
Cache the stable system prompt — Your system prompt (policy, escalation rules, refund limits, tone) is large and unchanging. Put a cache_control breakpoint on it so every turn after the first reads it at ~0.1× cost. Keep volatile content (the live order data, the timestamp) out of the cached prefix — see prompt-caching.
```
system = [{
    "type": "text",
    "text": SUPPORT_POLICY,            # frozen: policy, limits, escalation criteria
    "cache_control": {"type": "ephemeral"},
}]
```

Run the agentic loop — Standard tool loop: call the model, on stop_reason == "tool_use" execute each tool, append the assistant turn and the tool_result blocks (all results in ONE user message), and continue until end_turn. Cap iterations so a stuck agent can't loop forever.

import anthropic
client = anthropic.Anthropic()

messages = [{"role": "user", "content": user_msg}]
for _ in range(8):  # iteration cap
    resp = client.messages.create(
        model="claude-opus-4-8",
        max_tokens=4096,
        system=system,            # cached
        tools=TOOLS,
        messages=messages,
    )
    if resp.stop_reason != "tool_use":
        break
    messages.append({"role": "assistant", "content": resp.content})
    results = []
    for block in resp.content:
        if block.type == "tool_use":
            results.append({
                "type": "tool_result",
                "tool_use_id": block.id,
                "content": run_tool(block.name, block.input),  # may set is_error
            })
    messages.append({"role": "user", "content": results})

Emit a structured verdict — After the conversational turn, force a structured summary so downstream systems (logging, routing, dashboards) get reliable JSON. Use forced tool use — tool_choice pinned to a record_outcome tool — the most reliable structured-output path:

verdict = client.messages.create(
    model="claude-opus-4-8",
    max_tokens=512,
    system=system,
    messages=messages,
    tools=[{
        "name": "record_outcome",
        "description": "Record the structured outcome of this support interaction.",
        "input_schema": {
            "type": "object",
            "properties": {
                "resolution": {"type": "string", "enum": ["resolved", "refunded", "escalated", "needs_user"]},
                "refund_cents": {"type": "integer"},
                "policy_cited": {"type": "string"},
                "confidence": {"type": "number"},
            },
            "required": ["resolution", "confidence"],
            "additionalProperties": False,
        },
    }],
    tool_choice={"type": "tool", "name": "record_outcome"},  # forces the schema
)

Handle errors gracefully — Tool backends fail. When lookup_order 404s or times out, return a clear tool_result with is_error=True ("Order not found — ask the user to re-check the ID") so the model recovers in-conversation instead of crashing. Wrap the API loop in typed exception handling (RateLimitError, APIStatusError) with bounded retry.
Build an eval set — At least 10 scenarios with expected outcomes: a clean refund-after-confirm, a refund that must be denied (over policy limit), an unknown order ID (error recovery), an angry customer (escalate), a question answerable from policy alone (no tools). Run the agent over each and assert on the record_outcome verdict. This is your regression net — see prompt-evaluations.
Measure caching — Log usage.cache_read_input_tokens across a multi-turn run and confirm the policy prefix is being read from cache, not re-billed, after turn one.

Acceptance criteria — you're done when

Three tools exist, designed by reversibility: lookup_order (read-only), issue_refund (gated), escalate_to_human.
Refunds require a host-enforced confirmation gate — confirmed=true plus the host having seen the user confirm; an ungated refund returns an is_error result, not a charge.
The agent runs a bounded multi-turn tool loop and reads fresh order data rather than trusting stale context (Scenario 1 stale-data hazard).
Each interaction produces a structured verdict via forced tool_choice (record_outcome), validating against the schema.
Escalation fires on out-of-policy / low-confidence / dissatisfaction, citing a reason (routing + escalation, agentic-patterns).
Tool failures return is_error results the agent recovers from; API errors are caught with typed exceptions and bounded retry.
An eval set of ≥10 scenarios runs green, including a denied refund and an error-recovery case (prompt-evaluations).
Prompt caching on the stable policy prompt is verified via cache_read_input_tokens after turn one (prompt-caching).
You journaled which course concept was hardest to integrate in learning-journal-template and ticked the capstone in progress.

Stretch goals

Add a cheap Haiku intent classifier in front (billing / support / refund / escalate) — the Scenario 1 routing-first architecture — so the Opus agent runs a tighter prompt per branch.
Add provenance: have record_outcome cite the exact order ID and policy clause that drove the decision.
Add an evaluator-optimizer pass: a second model reviews the verdict against policy and bounces low-confidence refunds back for human review.
Move bulk eval runs onto the Batch API (Haiku) for the 50% discount.

Self-assessment rubric

Level	Signal
🟢 Got it	You can architect a production support agent end-to-end — gated tools, structured output, escalation, error recovery, evals, caching — and defend each design choice against the Scenario 1 pitfalls.
🟡 Almost	It works on the happy path, but the confirmation gate is model-trusted rather than host-enforced, or the eval set misses the denial / error cases.
🔴 Revisit	Refunds fire without a real gate, or there's no eval set. Re-read cca-scenarios (Scenario 1) and tool-design-principles, then rebuild the gate.