CodeSOTA's MMLU evidence includes zero-shot chain-of-thought, five-shot and Global MMLU Lite rows, so the benchmark name alone does not establish comparability.

Claim: C-0009
Verdict: supported
Verification: source verified
Evidence: 3 attached records
Authors: Kacper Wikiel

01 · Research context

What this claim answers.

Question: RQ-0004 · Which MMLU results are actually comparable?
Hypothesis: H-0008 · MMLU protocol explains part of score spread
Experiment: E-0006 · MMLU protocol provenance audit

02 · Evidence ledger

Inspect the rows beneath the sentence.

Current registry values are resolved live when available and compared with the captured claim snapshot. Provenance remains visible when resolution falls back to the snapshot.

benchmark-result:3005

GPT-4o on MMLU

source verified

Model: GPT-4o
Benchmark: MMLU
Metric: accuracy
Value: 87.2

Registry note records zero-shot chain-of-thought.

Registry note: Zero-shot CoT. gpt-4o-2024-05-13.

CodeSOTA benchmark_resultsobserved 2026-03-03unchanged since snapshotOpen registry evidence →Primary source ↗

benchmark-result:3483

Gemini 2.5 Pro on MMLU

CodeSOTA verified

Model: Gemini 2.5 Pro
Benchmark: MMLU
Metric: accuracy
Value: 89.8

Registry note records a Global MMLU Lite variant.

Registry note: backfilled 2026-04-23 from arxiv.org (2507.06261 Gemini 2.5 tech report, Global MMLU Lite variant)

CodeSOTA benchmark_resultsobserved 2026-03-27unchanged since snapshotOpen registry evidence →Primary source ↗

benchmark-result:3485

Llama 4 Maverick on MMLU

CodeSOTA verified

Model: Llama 4 Maverick
Benchmark: MMLU
Metric: accuracy
Value: 89.4

Registry note records a five-shot result.

Registry note: 5-shot. Source: Meta Llama 4 blog post and model card (April 2025).

CodeSOTA benchmark_resultsobserved 2026-03-27unchanged since snapshotOpen registry evidence →Primary source ↗

03 · Read at the right strength

Verification is not the same as reproduction.

source verified

“CodeSOTA verified” means the evidence row and its source have passed the registry’s verification path. It does not, by itself, mean an independent team reran the protocol. Reproduction is recorded separately in the claim verdict and experiment graph.

04 · Citation record

A stable public object.

Canonical URL: https://www.codesota.com/claims/C-0009
Created by: Kacper Wikiel
Created: 2026-08-20
Snapshot: 2026-08-20
API: /api/research/claims/C-0009