CodeSOTA · Experiment · E-0007Portable protocol · public evidence graph
Why this run exists

ARC-AGI-1 matched-model audit

Measure the visible score range for the same three frontier models on ARC-AGI-1.

Fork this experiment →Open RQ-0006
Experiment
E-0007
Status
completed
Outcome
supported
Forks
3 public or local starts recorded
Owner
Kacper Wikiel
01 · Design

Method before metrics.

Resolve the three verified benchmark_result rows live and compute max minus min. The five-point screening threshold was documented before interpreting the spread.

Hypothesis
H-0011 · ARC-AGI-1 produces at least a five-point range across o3, o4-mini and Gemini 2.5 Pro in the verified snapshot.
02 · Protocol

The comparison contract.

A reproduction matches this protocol. A fork changes it and declares the deviation.

dataset
arc-agi-1
controlled Variables
model set · metric
independent Variable
model
benchmarks
accuracy
screening threshold points
5
03 · Evidence outputs

What this experiment produced or evaluated.

Uses
benchmark:arc-agi-1
Contribute

Change the evidence, not just the discussion.

Start locally, inspect the portable manifest, and sign in only if you choose to publish.

Add evidence

Attach a run to this protocol.

Record the measurement and its provenance. New evidence is unverified and pending review until CodeSOTA checks it.

04 · Record

Stable, portable, attributable.

Stable ID
E-0007
Visibility
public
Created
2026-08-20
Updated
2026-08-20