Controlled 25M / 50M / 150M / 350M benchmark ladder

Measure where BPB, BLiMP, SciQ, ARC-Easy, LAMBADA and MMLU begin separating otherwise comparable base models.

01 · Design

Method before metrics.

Train a fixed model family at four scales, preserve the data/token ratio, and evaluate each checkpoint with the same harness and multiple seeds.

Question: RQ-0001 · Which evaluations provide useful signal for 25M–500M language models?
Hypothesis: H-0002 · SciQ provides above-chance separation at smaller parameter scales than standard MMLU.
Hypothesis: H-0003 · Standard MMLU accuracy remains too close to chance to be a primary KPI for base models below roughly 300M parameters.
Hypothesis: H-0010 · A useful benchmark progresses from leading indicator to primary KPI to control as model scale increases.

02 · Protocol

A reproduction matches this protocol. A fork changes it and declares the deviation.

model Family: decoder-only base language model
controlled Variables: tokenizer · dataset mixture · training token ratio · evaluation prompts
independent Variable: parameter count
seeds: 42 · 43 · 44
benchmarks: BPB · BLiMP · SciQ · ARC-Easy · LAMBADA · MMLU

03 · Evidence outputs

This planned experiment has not produced public evidence yet.

Contribute

Start locally, inspect the portable manifest, and sign in only if you choose to publish.

Add evidence

Record the measurement and its provenance. New evidence is unverified and pending review until CodeSOTA checks it.

04 · Record