Architecture comparison
Agentic complexity is not assumed to help — it's measured
Baseline 0 · no retrieval
A single LLM call, no retrieval at all. The control condition every other architecture must beat.
Baseline 1 · retrieve + synthesize
BM25-retrieve on the raw question, then one LLM call to synthesize an answer from the retrieved passages.
Agent 1 · planner + retrieve + synthesize
An LLM call first decomposes the question into 1-3 targeted search queries, then retrieves and synthesizes exactly as Baseline 1 does — isolating the planner's effect.
Baseline 1 vs. Agent 1, fully and fairly graded
| Metric | Baseline 1 (20/20) | Agent 1, worst-case (20/20) | Winner |
|---|---|---|---|
| Unsupported-claim rate | 0% | 6% | Baseline 1 |
| Claim support rate | 100% | 94% | Baseline 1 |
| Citation precision | 100% | 95% | Baseline 1 |
| Source validity rate | 100% | 95% | Baseline 1 |
| Abstention correct | 95% | 90% | Baseline 1 |
| Retrieval recall@k | 85% | 78% | Baseline 1 |
| Concept coverage | 40% | 45% | Agent 1 |
| Retrieval MRR | 57% | 65% | Agent 1 |
On task B4, Agent 1’s planner split a vague question into two sharper sub-queries and recovered a complete retrieval miss that Baseline 1 suffered (recall@k 0.0 → 1.0) — a genuine, mechanistic win for query planning.
On task D1, Agent 1’s own planner call returned syntactically invalid JSON (a real model error, not a parsing bug) and the task failed outright, while Baseline 1 answered it correctly.
Charging that one failure the worst-case value on every metric it would otherwise report — rather than letting it silently drop out of the denominator — reverses the naive read: Agent 1 loses on 6 of 8 shared metrics and wins only on concept coverage and retrieval ranking quality (MRR), at roughly 27% more latency. This number went through three honest revisions as two separate grading bugs were found and fixed; see methods for the full account.