Co-STORM: Human-in-the-Loop Discovery

Co-STORM (EMNLP 2024, arXiv 2408.15232) extends STORM from an offline article generator into a collaborative discovery interface. Where STORM runs its simulated conversations behind the scenes and hands you a finished article, Co-STORM puts the user inside the conversation.

The problem: unknown unknowns

Search engines and RAG chatbots share a structural limit: you only learn about what you thought to ask. If you don't know a sub-topic exists, you never query it. Co-STORM's design goal is to surface unknown unknowns — information the user didn't know to ask for — by letting them observe and steer a roundtable of LM experts instead of shouldering the entire questioning burden themselves. It is reverse prompting institutionalized: the system generates the questions, the human curates the direction.

Mechanics

Roundtable of opposing experts

Multiple CoStormExpert agents are instantiated with deliberately opposing stands on the topic. Disagreement is a feature: adversarial experts stress-test claims and surface facets a consensus panel would skip (compare multi-agent debate in agentic-patterns).

The Moderator — the serendipity engine

A Moderator agent watches the discourse. After N consecutive answering turns (conversation converging on one thread), it fires and injects a new discussion angle. Crucially, the angle is grounded in retrieved-but-uncited snippets — sources that came back from retrieval but that no expert has used yet. That reservoir of "found but unused" information is precisely where unknown unknowns live, which is why the Moderator is called the serendipity engine.

Warm start

Before the user sees anything, Co-STORM runs a background research pass so the roundtable begins already informed rather than cold-starting on generic questions.

Dynamic mind map

Every discourse turn is placed into a hierarchical mind map via embedding similarity plus LLM placement decisions, with periodic reorganization as the structure drifts. The map serves two roles at once:

1. Cognitive-load reducer — the user can see the conceptual territory at a glance and click into any node to steer.
2. Retrieval index — when generating the final report, the mind map organizes the collected information (a living cousin of the corpus in storm-pipeline Stage D; see also rag-patterns).

On-demand cited report

At any point the user can ask for a full cited report synthesized from the mind map and collected sources — the STORM article-generation machinery, invoked on an interactively-built knowledge base.

Interaction model

user turn (optional)          — inject a question or steer topic
expert turns                  — opposing experts ask/answer, grounded in retrieval
moderator turn (auto-fired)   — new angle from retrieved-but-uncited snippets
mind map update               — every turn placed; periodically reorganized

The user can stay passive (pure observation), fully active (asking everything), or anywhere between.

Evaluation

On WildSeek, a dataset of real complex information-seeking scenarios (topic + user goal), human evaluators preferred Co-STORM:

Comparison	Preference for Co-STORM
vs. search engines	70%
vs. RAG chatbots	78%

Key terms

• Co-STORM — collaborative discourse extension of STORM (EMNLP 2024) where a user observes and steers a roundtable of grounded LM experts to surface unknown unknowns.
• Unknown unknowns — information a user doesn't know exists and therefore can never query for; the failure mode of ask-driven interfaces Co-STORM targets.
• CoStormExpert — an LM expert agent in the roundtable, instantiated with a deliberately opposing stand to other experts.
• Moderator — the agent that fires after N consecutive answering turns and injects a new discussion angle grounded in retrieved-but-uncited snippets.
• Retrieved-but-uncited snippets — sources returned by retrieval that no expert has yet used; the Moderator's raw material and the system's serendipity reservoir.
• Warm start — the background research pass run before user interaction so the roundtable begins informed.
• Dynamic mind map — a hierarchical concept map built via embedding plus LLM placement and periodically reorganized; doubles as UI overview and retrieval index for report generation.
• WildSeek — the evaluation dataset of real complex information-seeking scenarios used to benchmark Co-STORM.

See also

• storm-overview / storm-pipeline — the underlying machinery
• reverse-prompting-overview — the interaction philosophy Co-STORM automates
• reverse-prompting-patterns — the Moderator as a reverse-prompting pattern