CodeSOTA research note · Polish replication · 15 August 2026
Dragon Hatchling,
trained in Polish
I trained a 25.3M-parameter BDH model end-to-end on 100M Polish byte tokens. This page documents what worked, how the architecture differs from a Transformer, and why Pathway’s ARC-AGI result makes the line of research worth watching.
SlayerLab / experiment 01
- 100M
- Polish tokens
- 10k
- steps
A working baseline,
not a frontier language model.
- 25.3M
- parameters
- 2,048
- sequence length
- 256
- UTF-8 byte vocabulary
- CC BY 4.0
- checkpoint license
01 / What I built
A Polish BDH baseline
you can inspect.
My goal was deliberately narrow: verify that the official BDH architecture can be trained and sampled end-to-end in Polish—not claim a new state of the art.
cleaned Polish byte tokens from polish-dynaword-mix; legal text capped near 7%
steps using ZClip and the paper’s B.2 protocol
random-initialization baseline to final validation loss
The model converged, loads from a standard 101 MB Safetensors checkpoint, and produces grammatically plausible Polish sentences directly from UTF-8 bytes—without an external tokenizer.
At 25M parameters and 100M training tokens, it is undertrained for high-quality generation. It has not been evaluated on ARC-AGI; the ARC result below belongs to Pathway’s separate, specialized 150M BDH-CQ model.
02 / The thesis
Not a tiny dragon.
A different coordinate system.
The paper calls the architecture Dragon Hatchling (BDH). “Baby Dragon Hatchling” is a commonly used expansion of the acronym, but not the paper’s formal title. Its central move is to describe one computation in two ways.
BDH-GPU
Dense tensor operations, linear attention, backpropagation and ordinary accelerators.
Graph BDH
Local messages among neurons; edge weights behave like changing synapses.
The authors construct a parameter- and state-size-preserving correspondence between these views. That bridge—not merely “attention plus ReLU”—is the paper’s real claim. [1]
BDH is a recurrent, linear-attention state-space model whose very wide positive activation space can be interpreted as neurons, while its fast-changing attention state can be interpreted as synapses.
03 / Architecture
One layer, unpacked.
BDH-GPU has a large “neuron” dimension n and a much smaller bottleneck d. Almost all learned weights live in three matrices: encoder E, and decoders Dₓ and Dᵧ. The weights are shared across layers. [2]
MEMORY WRITEρₜ,ₗ := (ρₜ₋₁,ₗ + LN(E yₜ,ₗ₋₁) xₜ,ₗᵀ) U
RESIDUAL PATHxₜ,ₗ := xₜ,ₗ₋₁ + ReLU(Dₓ LN(E yₜ,ₗ₋₁))
READ + GATEyₜ,ₗ := ReLU(Dᵧ LN(ρₜ₋₁,ₗ xₜ,ₗ)) ⊙ xₜ,ₗ
A readable transcription of Equation 8. U applies time decay/position handling; ⊙ is element-wise multiplication. Implementation variants move normalizations, so consult the paper and repository before reproducing results. [3]
Primary scaling axis; 32,768 in the paper’s 25M example.
Compact channel for communication; 256 in that example.
Repeated layers with shared weights; commonly 8–10 in examples.
04 / Working memory
The state changes
while it thinks.
The learned matrices are slow weights. The matrix ρ is fast state: every token adds an outer product, associating the current sparse activation with another activation. In the graph view, this is a synapse strengthening when two neurons co-activate—the Hebbian motif “fire together, wire together.”
concept activates
association added
memory changed
This is an architectural interpretation, not a claim that gradient pretraining itself is biologically realistic. The official model is trained end-to-end with ordinary backpropagation; Hebbian behavior describes the short-term inference state. [4]
Comparison
BDH and the Transformer.
This comparison is architectural. Similar validation loss does not imply equal downstream capability, training efficiency, inference speed, or quality under modern subword tokenization.
05 / Why this matters now
BDH moved the ARC-AGI
cost frontier.
On 11 August 2026, Pathway reported 29.5% pass@2 on public ARC-AGI-1 at an estimated $0.00070 per task with BDH-CQ, a specialized 150M-parameter model. An independent black-box evaluation reproduced the public-set result. Read their report ↗
public ARC-AGI-1, pass@2
per task on their assumptions
BDH-CQ, not BDH-25M-PL
I reproduced the architecture in Polish. Pathway produced the ARC-AGI result. They are connected evidence about the same model family, but they are not the same experiment or checkpoint.
06 / Evidence ledger
What was actually shown?
Transformer-like loss scaling
Across models from roughly 10M to 1B parameters, reported next-token losses track tuned GPT-2-style baselines on the authors’ language and translation setups.
Evidence: scaling experiments · paper §4.2 [5]Sparse positive activations
The paper reports roughly 5% activity in y vectors, varying by token, without an L1 sparsity penalty.
Evidence: activation analysis · paper §6.4 [6]Concept-linked synapses
Individual recovered synapse coordinates respond consistently to selected concepts across prompts, including some abstractions.
Evidence: probing on a translation model · paper §6.3Emergent graph structure
Thresholded encoder-decoder products show heavy tails and higher modularity than reported random baselines in a 24M model study.
Evidence: five seeds; detailed matrix analysis · paper §5.5 [7]Model merging by concatenation
Small translation models can be concatenated and retain partial behavior, but samples also mix languages and degrade.
Evidence: exploratory translation experiment · paper §7.106 / Interpretability
Three layers of visibility.
- Activation
Non-negative, sparse coordinates make “which neurons fired?” directly inspectable.
- Parameters
Products such as DₓE can be thresholded into a neuron-to-neuron communication graph.
- State
The changing attention state can be mapped back to synapse weights, exposing what the current context has strengthened.
A sparse coordinate or synapse is only useful after a careful causal and statistical validation. The paper provides compelling probes, not a complete interpretability benchmark against SAE or circuit-analysis baselines.
07 / Research frontier
Read the claims
at the right strength.
It is early-stage evidence
The paper is an arXiv v1 from September 2025. Treat its empirical and biological claims as research claims awaiting broad independent replication.
Loss is not capability
GPT-2-like loss scaling on selected corpora is narrower than modern evaluation across reasoning, retrieval, safety, and instruction following.
No free infinite context
State size is fixed, but distinguishable facts are capacity-limited and stale signals require damping. “No context window” does not mean perfect unbounded recall. [8]
Biological plausibility is scoped
The local inference rules admit a spiking-neuron interpretation. The released GPU model still uses dense tensors, LayerNorm and gradient-based training.
Efficiency needs benchmarks
Linear attention changes asymptotics, while wide states and repeated shared-weight layers create real FLOP and memory tradeoffs.
Replication surface is small
The official repository is intentionally compact. Reproducing the paper requires matching data, tokenizer, schedules, model variant, and evaluation details.
08 / Research kit
Start with primary sources.
01The formal definitions, proofs, figures and appendices.
Minimal PyTorch model and training entrypoint.
A gentler route into the ideas and visual intuition.
Models, Spaces, discussion, and follow-on experiments.
Suggested replication checklist +
- Record the exact paper/repository revision and architecture variant.
- Match tokenizer, corpus, token budget, optimizer, and parameter count.
- Report validation loss and wall-clock / FLOP / memory measurements.
- Compare against a tuned GPT-2 and at least one modern linear-attention SSM.
- Pre-register concept probes; include negative controls and multiple seeds.
- Test recall as sequence length, interference, and state decay vary.