PR #1394 · SP8192 + GPTQ Embeddings + Depth Recurrence + SDClip
The combined SP8192, GPTQ-embedding, depth-recurrence and SD-clip stack improves FineWeb validation BPB under the Parameter Golf budget.
Method before metrics.
On PR #1394: SP8192, GPTQ embeddings, looped layers 4-5, MuonEq-R, and std-based GPTQ clipping Declared changes include: Increase the vocabulary size from 4096 to 8192. GPTQ-quantize the embedding matrix instead of using simple round-to-nearest quantization. The other matrices were already using GPTQ. Remove the value embeddings. Replace the coprime-stride data loader from #726 with a simpler ShuffledSequenceLoader.
- Hypothesis
- H-PG-1394 · The combined SP8192, GPTQ-embedding, depth-recurrence and SD-clip stack improves FineWeb validation BPB under the Parameter Golf budget.
- Derived from
- 4096-Vocab + Larger Model + High WD + Simplifications ↗
external-run:ER-PG-1218 - Uses technique from
- Parallel Residuals + Mini Depth Recurrence ↗
external-run:ER-PG-1204 - Uses technique from
- Parameter Golf PR #1217 ↗
source:github-pr-1217 - Directly extended by
- PR #1413 · SP8192 + QK-Gain 5 + Legal Score-First TTT
experiment:E-PG-1413 - Directly extended by
- PR #1493 · SP8192 + 3-Layer Recurrence + Parallel Residuals + Legal TTT
experiment:E-PG-1493 - Technique reused by
- PR #1514 · SP8192 + Muon 0.97 + Legal Score-First TTT
experiment:E-PG-1514
The comparison contract.
A reproduction matches this protocol. A fork changes it and declares the deviation.
- model Family
- Parameter Golf language model
- dataset
- FineWeb validation
- benchmarks
- FineWeb validation BPB
- controlled Variables
- 16,000,000-byte artifact cap · 600-second training budget · 8×H100 SXM evaluation hardware · tokenizer-agnostic BPB
- independent Variable
- combined implementation changes declared in the source PR
- seeds
- 1 · 42 · 1234 · 1337 · 2025
- hardware
- 8×H100 SXM
- pr number
- 1,394
- source pr
- https://github.com/openai/parameter-golf/pull/1394
- source commit
- f4fa11fe7671036dd1e3b945e405750f21ddecfc
- source dataset commit
- f5c079314c4877fbb0af378c0abade5a8ca33d3a
- seed count
- 5
- metric direction
- lower_is_better
- artifact limit bytes
- 16,000,000
- training budget seconds
- 600
- declared changes
- Increase the vocabulary size from 4096 to 8192. · GPTQ-quantize the embedding matrix instead of using simple round-to-nearest quantization. The other matrices were already using GPTQ. · Remove the value embeddings. · Replace the coprime-stride data loader from #726 with a simpler ShuffledSequenceLoader. · Loop layers 4-5 twice (while sharing params): the idea is from #1204, but this script uses a simpler implementation and loops twice rather than once. · Use row-normalized Muon from #1217. · Choose the quantization clip threshold based on the standard deviation of the row rather than searching for a quantile with low reconstruction error. See the note below for motivation/details.
- declared relations
- [object Object] · [object Object] · [object Object]
What this experiment produced or evaluated.
- Produced
- SP8192 + GPTQ Embeddings + Depth Recurrence + SDClip ↗
external-run:ER-PG-1394 - Uses
- benchmark:fineweb-validation-bpb
Change the evidence, not just the discussion.
Start locally, inspect the portable manifest, and sign in only if you choose to publish.
Attach a run to this protocol.
Record the measurement and its provenance. New evidence is unverified and pending review until CodeSOTA checks it.
Stable, portable, attributable.
- Stable ID
- E-PG-1394
- Visibility
- public
- Created
- 2026-04-05
- Updated
- 2026-08-20