Complete evidence
60.4%
151/250 cases · 95% CI 54.2–66.3%
Original benchmark · MCRB-1Measured 2026-08-09
How much evidence survives a fixed context budget?
Six extractive selectors. 250 independently annotated questions. 136 full scientific papers. One auditable question: did at least one complete human evidence set survive?
MCRB-1 is a deliberately narrow retrieval benchmark. It measures exact evidence survival under a declared 2,048-token budget—not generated-answer quality.
Mean token reduction
74.4%
1,512 mean output tokens
Front truncation
25.6%
2.36× lower complete-set retention
Oracle ceiling
99.6%
39.2 points of ranking headroom
Frozen protocol
The labels came from people, not the system being tested.
The cohort is the first 250 eligible, answerable QASPER development questions ordered by SHA-256 of question ID. Every method receives the same fixed selection allowance. Exact human-highlighted spans determine retention; no LLM judge participates in the primary metric.
- Dataset
- QASPER v0.3.0 dev
- Mean input
- 6,447.1 BPE tokens
- Declared budget
- 2,048 tokens
- Primary metric
- Complete evidence set
Measured comparison
BM25 leads the deployable baselines—and still fails often.
The oracle uses known gold evidence and is an unattainable upper bound, not a competing product. Passage identifiers make traceability 100% for every extractive method by construction.
| Method | Complete | Any evidence | Mean recall | Reduction | p95 local |
|---|---|---|---|---|---|
| Maha BM25 | 60.4% | 72% | 66.2% | 74.4% | 5.89 ms |
| Maha keyword | 54% | 64% | 58.8% | 74.5% | 5.33 ms |
| Front truncation | 25.6% | 30.4% | 27.9% | 73.1% | 8.23 ms |
| Tail / recency | 20.4% | 25.2% | 22.7% | 73.2% | 7.92 ms |
| Seeded random | 21.6% | 33.2% | 26.9% | 73.3% | 8.16 ms |
| Oracle ceiling | 99.6% | 99.6% | 99.6% | 73.3% | 10.8 ms |
Failure analysis
The 99 incomplete cases split into two different observability problems.
A single aggregate retention rate hides whether ranking found some of the needed evidence or never reached it. Those failures call for different diagnostics.
151
Complete
At least one entire human-annotated evidence set survived selection.
29
Partial hit
Some annotated evidence survived, but no complete evidence set did.
70
Total miss
None of the annotated evidence appeared in the selected context.
Position breakdown
Location bias is reduced, not eliminated.
front evidence (95 cases)62.1% complete
middle evidence (93 cases)62.4% complete
back evidence (62 cases)54.8% complete
Front truncation retained 0% of complete evidence sets in both the middle and back buckets. BM25 retained 62.4% and 54.8%.
Capacity diagnosis
The budget could usually hold the evidence.
The gold-label oracle preserved a complete evidence set in 99.6% of cases under the same allowance. The 39.2-point gap is therefore primarily selection and ranking headroom, not proof that a larger context window is required.
That is the operational lesson: monitor no-hit, partial-hit, and complete-hit outcomes separately whenever evaluation labels exist.
Economics · workload-specific
Measured savings exceeded the machine fee.
- Mean input
- 6,447.1 tokens
- Tokens avoided
- 4,935.1
- Gross avoided
- $0.014805
- $0.001 multiple
- 14.80×
Uses a declared reference input rate of $3/million tokens. Output generation is excluded equally. This is not a universal savings promise.
Interpretation boundary
Retention is not answer quality.
MCRB-1 tests whether independently highlighted evidence remains available to a downstream model. It does not test whether that model reasons correctly, cites correctly, or tells the truth. Generative summarizers are excluded because exact-span scoring penalizes paraphrase, while an LLM judge would make the primary result model-dependent.
Reproduce and audit
Every case and measurement is public.
The cohort manifest contains IDs, hashes, token counts, and position buckets without source document text. The raw record file contains all 1,500 case-method measurements.
npm run benchmark:context-retentionDataset: QASPER v0.3.0 ↗, licensed CC BY 4.0. See the original paper ↗.
Related implementation notes