How do we minimize language-model BPB under a 16 MB artifact budget?
Which architecture, training, quantization and test-time adaptation choices minimize FineWeb validation bits per byte while the complete artifact remains under 16,000,000 bytes?
Why this question exists.
OpenAI Parameter Golf produced a dense public record of merged experiments, source PRs, declared predecessors, per-seed evidence and reproduction instructions. CodeSOTA imports the official leaderboard as a pinned research dataset.
Why it matters. This is a working model of cumulative empirical research: changes can be traced to their source, score progression can be separated from code ancestry, and every claim can retain the exact upstream evidence.
Keep the supported and refuted branches.
A negative result remains visible research memory. Each branch links its protocol and the claim the evidence supports or refutes.
The SP8192 recurrence stack improves parameter efficiency
The combined SP8192, GPTQ-embedding, depth-recurrence and SD-clip stack improves FineWeb validation BPB under the Parameter Golf budget.
QK gain and legal score-first TTT improve the SP8192 stack
Raising QK gain and adding legal score-first test-time training improves the PR #1394 stack under the same artifact and compute constraints.
Recurrence and parallel residual routing compound
The combined three-layer recurrence, parallel residual routing, QK-gain and legal TTT variant improves the PR #1394 base stack.
The Muon 0.97 and causal-tilt bundle improves its predecessor
The combined Muon 0.97 and causal token n-gram tilt variant improves the direct PR #1413 predecessor; the score delta is not attributed to either change alone.
The variable-length attention bundle moves the frontier
The combined variable-length attention, fused MLP and document-independent LoRA TTT variant improves its contemporary Parameter Golf base stack.
What the graph can currently say.
Merged Parameter Golf PR #1394 reports a 5-seed FineWeb validation result of 1.08563 BPB under the challenge constraints.
source verified1 evidence recordC-PG-1413Merged Parameter Golf PR #1413 reports a 3-seed FineWeb validation result of 1.08279 BPB under the challenge constraints.
source verified1 evidence recordC-PG-1493Merged Parameter Golf PR #1493 reports a 3-seed FineWeb validation result of 1.081 BPB under the challenge constraints.
source verified1 evidence recordC-PG-1514Merged Parameter Golf PR #1514 reports a 3-seed FineWeb validation result of 1.07983 BPB under the challenge constraints.
source verified1 evidence recordC-PG-1530Merged Parameter Golf PR #1530 reports a 3-seed FineWeb validation result of 1.07336 BPB under the challenge constraints.
source verified1 evidence recordFrontier progression is not the same thing as code lineage
The selected milestone scores descend from 1.08563 to 1.07336 BPB, but the declared parents branch: PR #1514 names #1413 as its direct predecessor, while PR #1530 derives from #1523 and incorporates work from #1354 and #1552. Adjacency on a leaderboard must not create a DERIVED_FROM edge.
Visible ownership.
- Stable ID
- RQ-PG-0001
- Created by
- CodeSOTA GitHub importer
- Contributors
- OpenAI Parameter Golf contributors · Kevin Clark · dexhunter · bigbag · samacqua
- Visibility
- public