What this benchmark does, and exactly where it stops

40 real PubMed abstracts (TP53 tumor-suppressor biology; BRCA1/BRCA2/PALB2 DNA repair and breast-cancer-risk biology; drug interactions, pathways, and bioinformatics methodology added for coverage), fetched via NCBI E-utilities and frozen with a retrieval timestamp. 20 gold tasks across 7 task types, each hand-constructed from a specific sentence of a specific abstract by a single researcher — labeled “researcher-constructed,” not expert-panel-derived.

Every metric on this site is computed from structured fields — retrieval sets, citation verification statuses, lexical token overlap — not from an LLM asked “is this good?” Citation support (direct / indirect / no support) is a lexical-overlap heuristic against a 50%/15% threshold, which will both miss paraphrased support and over-credit keyword-matching non-sequiturs. No LLM-as-judge grader exists yet.

A failed task used to drop out of every metric’s mean except a run-failure flag, making a run with failures look artificially comparable to one without. Fixed by charging failed tasks the worst-case value on every metric their architecture would otherwise report, reported alongside the plain (completion-conditional) mean.

Separately, a claim explaining an abstention (“the evidence doesn’t mention X”) was being scored as an uncited factual claim even when the model had already set a top-level abstained flag — double-penalizing the same failure. Fixed by keying off that structural flag instead of the claim’s own, sometimes-inconsistent self-label.

Both fixes changed a previously reported finding: Baseline1’s unsupported-claim rate on the 20-task set was reported as 5.26%, then retracted to 0.0% once the second bug was fixed. The Agent1-vs-Baseline1 comparison went through three revisions as a result — see the agents page for the final numbers, and treat any number quoted from this project as a claim with a run ID and a git SHA behind it, not a standalone fact.

LIVE — a real, non-mock model call. Currently a self-hosted Qwen3-family model via an OpenAI-compatible gateway, not Claude or GPT (no Anthropic key was available when these runs were made) — see every live-run page for this caveat, and do not read any single-model finding here as general LLM behavior.

SIMULATED — the offline mock provider, which has no world knowledge and can only echo back retrieved text verbatim. It exists to test the full pipeline for $0 with no API key, and its near-perfect citation precision is a property of its design, not evidence about real models.

FROZEN — the corpus and gold task set are versioned snapshots; the benchmark leaderboard never depends on a live LLM call to render.

A knowledge graph, an LLM-as-judge grader, calibration/reliability curves, systematic ablations, a cost-quality frontier analysis, and Agent2–4 (multi-branch retrieval, an evidence graph, and a researcher+verifier loop) are all explicitly not built. The /graph, /cost, and /paper pages say so directly rather than showing placeholder data.