QSTORM: Adapting STORM to Exam Questions

QSTORM is the adaptation of STORM this site uses to generate certification-parity practice questions. The insight transfers directly: just as article quality is bottlenecked on pre-writing, exam-item quality is bottlenecked on pre-itemwriting — blueprint analysis, misconception research, and item specification. QSTORM keeps STORM's persona-driven, retrieval-grounded skeleton (see storm-pipeline) but swaps every stage's deliverable from "article" to "validated item bank."

Stage 0 — Blueprint ingestion

Parse the exam blueprint into a hierarchy: domain → objective → testable skill. Then:

Item counts = domain weight × total item target (a 20%-weighted domain in a 90-item bank gets 18 items).
Assign a Bloom's-taxonomy distribution per objective (recall vs. application vs. analysis), so the bank isn't all definition-recall.

Stage 1 — Exam-writer personas

STORM's "Wikipedia editors" become item-writer archetypes:

Persona	Hunts for
Basic-fact examiner	Definitional, must-know claims
Practitioner / scenario writer	Realistic applied situations
Edge-case adversary	Limits, exceptions, boundary behavior
Misconception hunter	Documented wrong beliefs — raw material for distractors
Comparative analyst	X-vs-Y discriminations (e.g. when to choose which API)

The misconception hunter is the load-bearing addition: good distractors are not random wrong answers but attractive wrong answers, and attractiveness comes from real, documented misconceptions.

Stage 2 — Grounded conversations

Personas run STORM-style simulated conversations, but retrieval is restricted to an authoritative-docs-only corpus (official documentation, not the open web) — directly mitigating STORM's source-bias-transfer failure mode. Each conversation collects two typed artifacts:

(claim, citation) pairs — candidate keys and stems
(misconception, citation) pairs — candidate distractors

Stage 3 — Item-spec outline

STORM's article outline becomes an item-specification sheet: one spec per planned item mapping skill → Bloom level → claim(s) → misconception(s), with measurable blueprint recall (every blueprint skill must be covered by ≥1 spec; coverage is checked, not assumed).

Stage 4 — Item drafting (NBME-style rules)

Each item is drafted under standard item-writing constraints:

Cover-the-options test — the stem must be answerable with options hidden.
Homogeneous options — same grammatical form, length, and category.
Distractors must encode documented misconceptions from Stage 2.
No "all of the above" / "none of the above".
No cueing — no grammatical or length hints pointing at the key.

Stage 5 — Blocking verification

Automated checks that block an item from the bank on failure:

key entailment          — cited source must entail the keyed answer
distractor check        — each distractor falsifiable from sources,
                          yet plausible to a partial knower
single-best-answer      — blind model check: exactly one defensible answer
item-writing-flaws lint — cueing, heterogeneity, absolutes, etc.

Stage 6 — Difficulty calibration

A model-ensemble facility proxy: a panel of small/mid/large models answers each item blind; the fraction answering correctly approximates item facility (the human pass rate proxy). Items where a distractor attracts nobody ("dead distractors") get those distractors regenerated from fresh misconceptions.

Stage 7 — Assembly

Deduplicate near-identical items, rebalance against the blueprint (Stage 0 counts), and attach cited explanations to every option — reusing the (claim, citation) pairs so the study experience is grounded end to end. The assembled bank feeds the quiz engine described in how-this-site-works.

Key terms

QSTORM — STORM's pre-writing pipeline retargeted from articles to exam items: blueprint → personas → grounded research → item specs → drafted, verified, calibrated questions.
Exam blueprint — the official domain/objective/weight table for a certification; drives item counts and coverage checks.
Item facility — the fraction of examinees answering an item correctly; QSTORM proxies it with a blind model ensemble.
Distractor — an incorrect option; in QSTORM each must encode a documented misconception with a citation.
Misconception hunter — the persona whose sole job is farming (misconception, citation) pairs for distractor writing.
Cover-the-options test — item-writing rule that a stem must be answerable without seeing the options; failing it means the stem carries too little information.
Homogeneous options — options matched in form, length, and category so no option stands out structurally.
Cueing — unintended hints (grammar agreement, longest-option-is-correct) that let testwise examinees find the key without knowledge; linted in Stage 5.
Dead distractor — a distractor no examinee (or ensemble model) ever selects; detected in calibration and regenerated.
Key entailment — verification that the cited source logically entails the keyed answer, not merely mentions its topic.