Project 07 — Structured Extraction from Messy Text

Enforces: reliable structured output — a forced-tool JSON schema, handling nullable/missing fields, and a validation step that re-prompts on schema failure (from building-with-the-claude-api) Surface: code · Time: ~75 min · Difficulty: 🔴 stretch

Why this project

"Extract the data as JSON" works until it doesn't — a stray prose preamble, a hallucinated field, a missing value rendered as the string "N/A". Production extraction needs a guaranteed shape and a recovery path. This project builds both: a forced-tool schema that constrains the output, and a validate-then-re-prompt loop that catches what slips through.

What you'll build

An extractor that pulls structured JSON from messy text (invoices or emails) using a forced tool, handles nullable and missing fields correctly, and re-prompts on validation failure before giving up.

Steps

Gather messy inputs — collect 3–5 real-ish blobs: a pasted invoice, a rambling order email, a signature block. Variety matters — include one that's missing a field you ask for.

Design the schema — define the target shape with explicit types, mark genuinely optional fields nullable, and list only the truly-required ones in required:

extract_tool = {
    "name": "record_invoice",
    "description": "Record the structured fields extracted from an invoice or order email.",
    "input_schema": {
        "type": "object",
        "properties": {
            "vendor":      {"type": "string"},
            "invoice_id":  {"type": "string"},
            "total":       {"type": "number"},
            "currency":    {"type": "string", "enum": ["USD", "EUR", "GBP"]},
            "due_date":    {"type": ["string", "null"], "description": "ISO date, or null if absent"},
            "line_items":  {"type": "array", "items": {"type": "string"}},
        },
        "required": ["vendor", "total", "currency"],
    },
}

Force the tool — set tool_choice so the model must return the schema (no prose, no skipping). The structured payload comes back in the tool_use block's input:

resp = client.messages.create(
    model="claude-opus-4-8", max_tokens=1024,
    tools=[extract_tool],
    tool_choice={"type": "tool", "name": "record_invoice"},
    system="Extract fields exactly. If a field is absent, use null — never invent a value.",
    messages=[{"role": "user", "content": f"<document>\n{blob}\n</document>"}],
)
data = next(b.input for b in resp.content if b.type == "tool_use")

Handle nullable / missing fields explicitly — instruct the model (as above) that absent fields are null, not guessed and not the string "N/A". Run your missing-field input and confirm due_date comes back null, not fabricated.

Add a validation step — validate data against the schema (use jsonschema, or a Pydantic model). On failure, re-prompt once: send the model its own bad output plus the validation error and ask it to fix it. Cap retries so you can't loop forever:

from jsonschema import validate, ValidationError

def extract(blob, retries=1):
    messages = [{"role": "user", "content": f"<document>\n{blob}\n</document>"}]
    for attempt in range(retries + 1):
        resp = client.messages.create(
            model="claude-opus-4-8", max_tokens=1024,
            tools=[extract_tool],
            tool_choice={"type": "tool", "name": "record_invoice"},
            messages=messages,
        )
        data = next(b.input for b in resp.content if b.type == "tool_use")
        try:
            validate(data, extract_tool["input_schema"])
            return data
        except ValidationError as e:
            messages.append({"role": "assistant", "content": resp.content})
            messages.append({"role": "user", "content": [{
                "type": "tool_result",
                "tool_use_id": next(b.id for b in resp.content if b.type == "tool_use"),
                "content": f"Schema validation failed: {e.message}. Return corrected fields.",
                "is_error": True,
            }]})
    raise ValueError("extraction failed after retries")

Run the batch and tally — run all inputs, count clean-on-first-try vs needed-a-retry vs failed. That tally is your reliability signal.

Acceptance criteria — you're done when

Your schema marks optional fields nullable and lists only truly-required fields in required.
You force the tool with tool_choice and read the payload from the tool_use block's input.
A missing-field input yields null for that field — not an invented or "N/A" value.
A validation failure triggers exactly one re-prompt with the error fed back, then gives up.
You ran all inputs and tallied first-try / retry / failed counts.
You journaled which input broke your schema and how you handled it in learning-journal-template.

Stretch goals

Replace the hand-written schema with client.messages.parse(...) and a Pydantic model (structured outputs) and compare reliability.
Add a confidence field and have the model flag low-confidence extractions for human review.
Extract from a 10-page PDF by passing it as a document content block instead of pasted text.

Self-assessment rubric

Level	Signal
🟢 Got it	Your extractor returns a guaranteed shape, handles missing data, and self-corrects on failure.
🟡 Almost	Extraction works on clean inputs but missing fields or validation recovery are shaky.
🔴 Revisit	Output is still free-form or invents values; re-read the structured-output material in building-with-the-claude-api.