CodeSOTA · Experiment · E-0006Portable protocol · public evidence graph
MMLU protocol provenance audit
Partition registry results into comparable protocol families before making cross-model claims.
01 · Design
Method before metrics.
Read source notes and primary references, then group rows by benchmark variant, shot count, reasoning mode and model version.
- Hypothesis
- H-0008 · Shot count, reasoning mode and benchmark variant explain a material part of differences published as MMLU accuracy.
02 · Protocol
The comparison contract.
A reproduction matches this protocol. A fork changes it and declares the deviation.
- dataset
- mmlu
- controlled Variables
- benchmark variant · shot count · reasoning mode · model version
- independent Variable
- model
- benchmarks
- MMLU accuracy
03 · Evidence outputs
What this experiment produced or evaluated.
This planned experiment has not produced public evidence yet.
Contribute
Change the evidence, not just the discussion.
Start locally, inspect the portable manifest, and sign in only if you choose to publish.
Add evidence
Attach a run to this protocol.
Record the measurement and its provenance. New evidence is unverified and pending review until CodeSOTA checks it.
04 · Record
Stable, portable, attributable.
- Stable ID
- E-0006
- Visibility
- public
- Created
- 2026-08-20
- Updated
- 2026-08-20