CodeSOTA · Research question · RQ-0004Updated 2026-08-20
What we want to learn

Which MMLU results are actually comparable?

How much of the visible MMLU spread comes from model quality, and how much comes from prompt, shot count, reasoning mode or benchmark variant?

Fork the next experiment →Inspect evidence
Status
active
Evidence
moderate evidence
Graph
2 hypotheses · 1 experiments
Claims
1 public · 3 attachments
Reproductions
0 recorded
Current finding

Protocol heterogeneity is directly visible in current CodeSOTA evidence; a normalized comparison set remains open work.

01 · Context

Why this question exists.

The registry preserves source notes for MMLU rows. Those notes include materially different protocols under a familiar benchmark name.

Why it matters. A leaderboard row without its protocol can turn an evidence registry into a false precision machine.

knowledge and reasoning →commonsense reasoning →
02 · Research tree

Keep the supported and refuted branches.

A negative result remains visible research memory. Each branch links its protocol and the claim the evidence supports or refutes.

H-0008testing

MMLU protocol explains part of score spread

Shot count, reasoning mode and benchmark variant explain a material part of differences published as MMLU accuracy.

E-0006MMLU protocol provenance auditpendingC-0009CodeSOTA's MMLU evidence includes zero-shot chain-of-thought, five-shot and Global MMLU Lite rows, so the benchmark name alone does not establish comparability.
H-0009proposed

Claim snapshots prevent history rewrites

Snapshot-aware evidence makes later benchmark corrections legible without silently changing the historical claim.

03 · Claims and findings

What the graph can currently say.

C-0009

CodeSOTA's MMLU evidence includes zero-shot chain-of-thought, five-shot and Global MMLU Lite rows, so the benchmark name alone does not establish comparability.

source verified3 evidence records
F-0004 · investigating

Leaderboard rows need protocol context

MMLU provenance already records different evaluation families. The next step is to expose that structure beside every derived claim.

04 · Provenance

Visible ownership.

Stable ID
RQ-0004
Created by
Kacper Wikiel
Contributors
Kacper Wikiel · CodeSOTA registry contributors
Visibility
public