CodeSOTA · Research question · RQ-0006Updated 2026-08-20
What we want to learn

Which reasoning benchmarks still separate frontier models?

Under CodeSOTA's currently verified evidence, which public reasoning benchmarks still separate the same frontier model set?

Fork the next experiment →Inspect evidence
Status
partially answered
Evidence
moderate evidence
Graph
3 hypotheses · 3 experiments
Claims
3 public · 9 attachments
Reproductions
0 recorded
Current finding

In this matched snapshot ARC-AGI-1 spans 31.4 points, ARC-Challenge spans 0.8, and GSM8K spans 0. Protocol-matched reruns are still required before calling this an intrinsic benchmark property.

01 · Context

Why this question exists.

The same three models—o3, o4-mini and Gemini 2.5 Pro—have verified registry rows on ARC-AGI-1, ARC-Challenge and GSM8K. This matched snapshot lets us inspect score spread without silently changing the model set.

Why it matters. A saturated benchmark consumes evaluation effort without helping choose among frontier systems. A visible negative result is useful research memory.

reasoning →logical reasoning →commonsense reasoning →mathematical reasoning →
02 · Research tree

Keep the supported and refuted branches.

A negative result remains visible research memory. Each branch links its protocol and the claim the evidence supports or refutes.

H-0011supported

ARC-AGI-1 separates the matched frontier snapshot

ARC-AGI-1 produces at least a five-point range across o3, o4-mini and Gemini 2.5 Pro in the verified snapshot.

E-0007ARC-AGI-1 matched-model auditsupportedC-0011ARC-AGI-1 spans 31.4 percentage points across the matched o3, o4-mini and Gemini 2.5 Pro CodeSOTA snapshot.
H-0012refuted

ARC-Challenge separates the matched frontier snapshot

ARC-Challenge produces at least a five-point range across o3, o4-mini and Gemini 2.5 Pro in the verified snapshot.

E-0008ARC-Challenge matched-model auditrefutedC-0012ARC-Challenge spans only 0.8 percentage points across the matched o3, o4-mini and Gemini 2.5 Pro CodeSOTA snapshot.
H-0013refuted

GSM8K separates the matched frontier snapshot

GSM8K produces at least a five-point range across o3, o4-mini and Gemini 2.5 Pro in the verified snapshot.

E-0009GSM8K matched-model auditrefutedC-0013GSM8K reports 99.0 for all three models in the matched o3, o4-mini and Gemini 2.5 Pro CodeSOTA snapshot.
03 · Claims and findings

What the graph can currently say.

C-0011

ARC-AGI-1 spans 31.4 percentage points across the matched o3, o4-mini and Gemini 2.5 Pro CodeSOTA snapshot.

CodeSOTA verified3 evidence recordsC-0012

ARC-Challenge spans only 0.8 percentage points across the matched o3, o4-mini and Gemini 2.5 Pro CodeSOTA snapshot.

CodeSOTA verified3 evidence recordsC-0013

GSM8K reports 99.0 for all three models in the matched o3, o4-mini and Gemini 2.5 Pro CodeSOTA snapshot.

CodeSOTA verified3 evidence records
F-0005 · confirmed

Frontier reasoning benchmarks have different visible resolution

For the same three-model snapshot, ARC-AGI-1 separates scores substantially more than ARC-Challenge or GSM8K. This is a descriptive registry finding, not yet a causal claim about benchmark quality.

04 · Provenance

Visible ownership.

Stable ID
RQ-0006
Created by
Kacper Wikiel
Contributors
Kacper Wikiel · CodeSOTA registry contributors
Visibility
public