PR #1530 · VarLen Attention + Fused MLP + Doc-Independent Legal TTT
The combined variable-length attention, fused MLP and document-independent LoRA TTT variant improves its contemporary Parameter Golf base stack.
Method before metrics.
On PR #1530: variable-length FA3 attention, fused Triton MLP, grouped small-parameter all-reduces, and doc-independent score-first LoRA TTT Declared changes include: Removes the need for the model to learn to ignore pre-BOS content from unrelated documents Reduces wasted FLOPs: e.g. 10 short (100-token) docs packed into a 1k-token buffer cost proportional to 100 1002 = 1M attention FLOPs vs 10 10002 = 10M with dense attention.
- Hypothesis
- H-PG-1530 · The combined variable-length attention, fused MLP and document-independent LoRA TTT variant improves its contemporary Parameter Golf base stack.
- Derived from
- Parameter Golf PR #1523 ↗
source:github-pr-1523 - Uses technique from
- Parameter Golf PR #1552 ↗
source:github-pr-1552 - Uses technique from
- Parameter Golf PR #1354 ↗
source:github-pr-1354 - Compared with
- Improved Parallel Residuals + CUTLASS EVT + Legal TTT ↗
external-run:ER-PG-1529 - Compared with
- PR #1493 · SP8192 + 3-Layer Recurrence + Parallel Residuals + Legal TTT
experiment:E-PG-1493
The comparison contract.
A reproduction matches this protocol. A fork changes it and declares the deviation.
- model Family
- Parameter Golf language model
- dataset
- FineWeb validation
- benchmarks
- FineWeb validation BPB
- controlled Variables
- 16,000,000-byte artifact cap · 600-second training budget · 8×H100 SXM evaluation hardware · tokenizer-agnostic BPB
- independent Variable
- combined implementation changes declared in the source PR
- seeds
- 0 · 1 · 2
- hardware
- 8×H100 SXM
- pr number
- 1,530
- source pr
- https://github.com/openai/parameter-golf/pull/1530
- source commit
- 7dca3ded46d2d6f537fda34777e5d0cad47f9232
- source dataset commit
- f5c079314c4877fbb0af378c0abade5a8ca33d3a
- seed count
- 3
- metric direction
- lower_is_better
- artifact limit bytes
- 16,000,000
- training budget seconds
- 600
- declared changes
- Removes the need for the model to learn to ignore pre-BOS content from unrelated documents · Reduces wasted FLOPs: e.g. 10 short (100-token) docs packed into a 1k-token buffer cost proportional to 100 1002 = 1M attention FLOPs vs 10 10002 = 10M with dense attention.
- declared relations
- [object Object] · [object Object] · [object Object] · [object Object] · [object Object]
What this experiment produced or evaluated.
- Produced
- VarLen Attention + Fused MLP + Doc-Independent Legal TTT ↗
external-run:ER-PG-1530 - Uses
- benchmark:fineweb-validation-bpb
Change the evidence, not just the discussion.
Start locally, inspect the portable manifest, and sign in only if you choose to publish.
Attach a run to this protocol.
Record the measurement and its provenance. New evidence is unverified and pending review until CodeSOTA checks it.
Stable, portable, attributable.
- Stable ID
- E-PG-1530
- Visibility
- public
- Created
- 2026-04-11
- Updated
- 2026-08-20