Which evaluations provide useful signal for 25M–500M language models?
Which evaluation metrics provide non-saturated, reproducible learning signal for base language models between 25M and 500M parameters?
Why this question exists.
Large-model leaderboards often collapse small base models to chance. The Slayer small-model programme needs a ladder that exposes progress before MMLU-style accuracy becomes informative.
Why it matters. A scale-appropriate suite can decide which checkpoints deserve more compute and can prevent a genuine training improvement from being mistaken for no progress.
Keep the supported and refuted branches.
A negative result remains visible research memory. Each branch links its protocol and the claim the evidence supports or refutes.
Byte-level loss is an early signal
Validation bits-per-byte changes smoothly before downstream exact-match benchmarks separate 25M-class models.
SciQ separates small models before MMLU
SciQ provides above-chance separation at smaller parameter scales than standard MMLU.
MMLU is floor-limited below 300M
Standard MMLU accuracy remains too close to chance to be a primary KPI for base models below roughly 300M parameters.
What the graph can currently say.
The BDH-25M-PL run reduced validation loss from an approximately 5.6 random baseline to 1.41 after 10,000 steps on 100M Polish byte tokens.
source verified1 evidence recordC-0002BDH-25M-PL has not been evaluated on ARC-AGI; Pathway's reported ARC-AGI result belongs to a separate 150M BDH-CQ system.
source verified1 evidence recordTraining signal precedes capability evidence
The 25M checkpoint demonstrates convergence, but it does not yet answer which downstream evaluations are informative at that scale.
Visible ownership.
- Stable ID
- RQ-0001
- Created by
- Kacper Wikiel
- Contributors
- Kacper Wikiel
- Visibility
- public