Benchmark leaderboard
No single opaque score — every metric shown separately
Cost and quality are never blended into one number. “Worst-case” rows charge a failed task the worst possible value on every metric its architecture would otherwise report, instead of silently dropping it from that metric’s denominator — see methods for why that distinction mattered in practice.
Canonical live run
live-gateway-v1-agent1-001
bra-corpus-v1 · 20 tasks · openai-compatible
| Architecture | Concept coverage | Evidence recall@k | Claim support | Citation precision | Unsupported-claim rate | Abstention correct | Failures | Latency |
|---|---|---|---|---|---|---|---|---|
| Baseline 0 · no retrieval | 18% | — | 0% | — | 100% | 50% | 0/20 | 74.4s |
| Baseline 1 · retrieve + synthesize | 40% | 85% | 100% | 100% | 0% | 95% | 0/20 | 126.9s |
| Agent 1 · planner + retrieve + synthesize | 48% worst-case 45% | 82% worst-case 78% | 100% worst-case 94% | 100% worst-case 95% | 0% worst-case 6% | 95% worst-case 90% | 1/20 | 160.8s |
Offline mock (pipeline-correctness only — see methods)
smoke-v1-agent1-001
bra-corpus-v1 · 20 tasks · mock-extractive-v0
| Architecture | Concept coverage | Evidence recall@k | Claim support | Citation precision | Unsupported-claim rate | Abstention correct | Failures | Latency |
|---|---|---|---|---|---|---|---|---|
| Baseline 0 · no retrieval | 10% | — | — | — | — | 20% | 0/20 | 0.0s |
| Baseline 1 · retrieve + synthesize | 39% | 85% | 100% | 100% | 0% | 80% | 0/20 | 0.0s |
| Agent 1 · planner + retrieve + synthesize | 39% | 85% | 100% | 100% | 0% | 80% | 0/20 | 0.0s |
4 earlier runs (v0 corpus, fewer architectures)
live-gateway-001
bra-corpus-v0 · 10 tasks · openai-compatible
| Architecture | Concept coverage | Evidence recall@k | Claim support | Citation precision | Unsupported-claim rate | Abstention correct | Failures | Latency |
|---|---|---|---|---|---|---|---|---|
| Baseline 0 · no retrieval | 16% | — | 0% | — | 100% | 50% | 0/10 | 33.9s |
| Baseline 1 · retrieve + synthesize | 43% | 85% | 100% | 100% | 0% | 100% | 0/10 | 52.6s |
live-gateway-v1-001
bra-corpus-v1 · 20 tasks · openai-compatible
| Architecture | Concept coverage | Evidence recall@k | Claim support | Citation precision | Unsupported-claim rate | Abstention correct | Failures | Latency |
|---|---|---|---|---|---|---|---|---|
| Baseline 0 · no retrieval | 18% | — | 0% | — | 100% | 50% | 0/20 | 73.6s |
| Baseline 1 · retrieve + synthesize | 40% | 85% | 100% | 100% | 0% | 90% | 0/20 | 115.8s |
smoke-001
bra-corpus-v0 · 10 tasks · mock-extractive-v0
| Architecture | Concept coverage | Evidence recall@k | Claim support | Citation precision | Unsupported-claim rate | Abstention correct | Failures | Latency |
|---|---|---|---|---|---|---|---|---|
| Baseline 0 · no retrieval | 10% | — | — | — | — | 20% | 0/10 | 0.0s |
| Baseline 1 · retrieve + synthesize | 41% | 85% | 100% | 100% | 0% | 80% | 0/10 | 0.0s |
smoke-v1-001
bra-corpus-v1 · 20 tasks · mock-extractive-v0
| Architecture | Concept coverage | Evidence recall@k | Claim support | Citation precision | Unsupported-claim rate | Abstention correct | Failures | Latency |
|---|---|---|---|---|---|---|---|---|
| Baseline 0 · no retrieval | 10% | — | — | — | — | 20% | 0/20 | 0.0s |
| Baseline 1 · retrieve + synthesize | 39% | 85% | 100% | 100% | 0% | 80% | 0/20 | 0.0s |