CodeSOTA · Clickable claim · C-0012Evidence snapshot 2026-08-20
Public evidence statement

ARC-Challenge spans only 0.8 percentage points across the matched o3, o4-mini and Gemini 2.5 Pro CodeSOTA snapshot.

Claim
C-0012
Verdict
supported
Verification
CodeSOTA verified
Evidence
3 attached records
Authors
Kacper Wikiel
01 · Research context

What this claim answers.

Hypothesis
H-0012 · ARC-Challenge separates the matched frontier snapshot
02 · Evidence ledger

Inspect the rows beneath the sentence.

Current registry values are resolved live when available and compared with the captured claim snapshot. Provenance remains visible when resolution falls back to the snapshot.

benchmark-result:3515

o3 on ARC-Challenge

CodeSOTA verified
Model
o3
Benchmark
ARC-Challenge
Metric
accuracy
Value
98.1

Verified o3 ARC-Challenge row.

Registry note: 0-shot. Source: OpenAI simple-evals (2025).

CodeSOTA benchmark_resultsobserved 2026-03-27unchanged since snapshotOpen registry evidence →Primary source ↗
benchmark-result:3516

o4-mini on ARC-Challenge

CodeSOTA verified
Model
o4-mini
Benchmark
ARC-Challenge
Metric
accuracy
Value
97.3

Verified o4-mini ARC-Challenge row.

Registry note: 0-shot. Source: OpenAI simple-evals (2025).

CodeSOTA benchmark_resultsobserved 2026-03-27unchanged since snapshotOpen registry evidence →Primary source ↗
benchmark-result:3517

Gemini 2.5 Pro on ARC-Challenge

CodeSOTA verified
Model
Gemini 2.5 Pro
Benchmark
ARC-Challenge
Metric
accuracy
Value
97.8

Verified Gemini 2.5 Pro ARC-Challenge row.

Registry note: 0-shot CoT. Source: Gemini 2.5 Pro technical report (April 2025).

CodeSOTA benchmark_resultsobserved 2026-03-27unchanged since snapshotOpen registry evidence →Primary source ↗
03 · Read at the right strength

Verification is not the same as reproduction.

codesota verified

“CodeSOTA verified” means the evidence row and its source have passed the registry’s verification path. It does not, by itself, mean an independent team reran the protocol. Reproduction is recorded separately in the claim verdict and experiment graph.

04 · Citation record

A stable public object.

Created by
Kacper Wikiel
Created
2026-08-20
Snapshot
2026-08-20