PR #1413 · SP8192 + QK-Gain 5 + Legal Score-First TTT
Raising QK gain and adding legal score-first test-time training improves the PR #1394 stack under the same artifact and compute constraints.
Method before metrics.
On PR #1413: QK-Gain 5.0 + legal score-first TTT on the PR #1394 SP8192 stack Declared changes include: No SLOT, no pre-quant TTT, no ETLB, no n-gram cache, no tokenizer change. Score-first TTT follows the PR #549 precedent — every chunk is scored under inferencemode() before any parameter update. Same base stack as PR #1394: sp8192 BPE, 11L/512d/8H/4KV, MLP 4×, Partial RoPE 16d, depth recurrence (loop layers 4–5 twice from 50% training), MuonEq-R WD=0.085, full-Hessian GPTQ int6 + int8 embeddings + SD-clip, Brotli+byte-shuffle, EMA. Two changes: (1) QKGAININIT raised from 4.0 → 5.0; (2) added a legal score-first TTT sliding pass (LR=0.005, 3 epochs, freezeblocks=0) as an additional eval mode. Condition 3 (Score before update): Every TTT chunk is scored under torch.inferencemode() BEFORE any parameter update. Training on a chunk only happens AFTER its scoring has been accumulated into losssum. Matches the PR #549 pattern.
- Hypothesis
- H-PG-1413 · Raising QK gain and adding legal score-first test-time training improves the PR #1394 stack under the same artifact and compute constraints.
- Derived from
- PR #1394 · SP8192 + GPTQ Embeddings + Depth Recurrence + SDClip
experiment:E-PG-1394 - Directly extended by
- PR #1514 · SP8192 + Muon 0.97 + Legal Score-First TTT
experiment:E-PG-1514 - Technique reused by
- PR #1493 · SP8192 + 3-Layer Recurrence + Parallel Residuals + Legal TTT
experiment:E-PG-1493
The comparison contract.
A reproduction matches this protocol. A fork changes it and declares the deviation.
- model Family
- Parameter Golf language model
- dataset
- FineWeb validation
- benchmarks
- FineWeb validation BPB
- controlled Variables
- 16,000,000-byte artifact cap · 600-second training budget · 8×H100 SXM evaluation hardware · tokenizer-agnostic BPB
- independent Variable
- combined implementation changes declared in the source PR
- seeds
- 0 · 42 · 1234
- hardware
- 8×H100 SXM
- pr number
- 1,413
- source pr
- https://github.com/openai/parameter-golf/pull/1413
- source commit
- cd04a8b7b57380ea48de4e195ebb65ee27076313
- source dataset commit
- f5c079314c4877fbb0af378c0abade5a8ca33d3a
- seed count
- 3
- metric direction
- lower_is_better
- artifact limit bytes
- 16,000,000
- training budget seconds
- 600
- declared changes
- No SLOT, no pre-quant TTT, no ETLB, no n-gram cache, no tokenizer change. Score-first TTT follows the PR #549 precedent — every chunk is scored under inferencemode() before any parameter update. · Same base stack as PR #1394: sp8192 BPE, 11L/512d/8H/4KV, MLP 4×, Partial RoPE 16d, depth recurrence (loop layers 4–5 twice from 50% training), MuonEq-R WD=0.085, full-Hessian GPTQ int6 + int8 embeddings + SD-clip, Brotli+byte-shuffle, EMA. · Two changes: (1) QKGAININIT raised from 4.0 → 5.0; (2) added a legal score-first TTT sliding pass (LR=0.005, 3 epochs, freezeblocks=0) as an additional eval mode. · Condition 3 (Score before update): Every TTT chunk is scored under torch.inferencemode() BEFORE any parameter update. Training on a chunk only happens AFTER its scoring has been accumulated into losssum. Matches the PR #549 pattern. · No pre-quant TTT on val data. · No tokenizer change — uses PR #1394's SentencePiece BPE 8192 unchanged.
- declared relations
- [object Object]
What this experiment produced or evaluated.
- Produced
- SP8192 + QK-Gain 5 + Legal Score-First TTT ↗
external-run:ER-PG-1413 - Uses
- benchmark:fineweb-validation-bpb
Change the evidence, not just the discussion.
Start locally, inspect the portable manifest, and sign in only if you choose to publish.
Attach a run to this protocol.
Record the measurement and its provenance. New evidence is unverified and pending review until CodeSOTA checks it.
Stable, portable, attributable.
- Stable ID
- E-PG-1413
- Visibility
- public
- Created
- 2026-04-06
- Updated
- 2026-08-20