The published CodeSOTA ladder shows SciQ separating earlier than ARC-Challenge across the Pythia 70M–6.9B reference series.

Claim: C-0010
Verdict: supported
Verification: source verified
Evidence: 1 attached record
Authors: Kacper Wikiel

01 · Research context

What this claim answers.

Question: RQ-0005 · When should a benchmark move from leading to primary to control?
Hypothesis: H-0010 · Benchmark roles move with scale
Experiment: E-0002 · Controlled 25M / 50M / 150M / 350M benchmark ladder

02 · Evidence ledger

Inspect the rows beneath the sentence.

Current registry values are resolved live when available and compared with the captured claim snapshot. Provenance remains visible when resolution falls back to the snapshot.

external-run:ER-0005

Pythia benchmark dynamic-range reference

source verified

sciq 70m 60.1sciq 6 9b 89.7arc challenge 70m 18.1arc challenge 6 9b 31.3

Published reference series for dynamic-range comparison.

Registry note: Published CodeSOTA methodology note using Pythia reference scores to illustrate overlapping benchmark ranges.

codesotaobserved 2026-08-16Open registry evidence →

03 · Read at the right strength

Verification is not the same as reproduction.

source verified

“CodeSOTA verified” means the evidence row and its source have passed the registry’s verification path. It does not, by itself, mean an independent team reran the protocol. Reproduction is recorded separately in the claim verdict and experiment graph.

04 · Citation record

A stable public object.

Canonical URL: https://www.codesota.com/claims/C-0010
Created by: Kacper Wikiel
Created: 2026-08-20
Snapshot: 2026-08-20
API: /api/research/claims/C-0010