Checked August 2026
Interactive transformer laboratory
A transformer,
under glass.
Inspect a tiny decoder model closely enough to see every number. The matrices are live, the probabilities sum to one, the gradient step is real, and every memory estimate comes from the architecture you choose.
- Teaching model
- 4 tokens · 4 dimensions · 2 projection heads
- What is simplified
- Scale, vocabulary, and the isolated training update
- What is not
- The equations, masks, softmax, gradients, sampling, or cache math
Chapter 01
Prompt → vectors
xᵢ = √d_model · E[tokenᵢ] + PEᵢThe prompt becomes an ordered matrix
system instructionexamplesretrieved evidenceuser task
Scaled embedding √d · E[rates]
+[1.33, -1.13, 0.86, -0.52]Position vector P3
=[0.14, -0.99, 0.03, 1.00]Residual stream x3
[1.47, -2.12, 0.89, 0.48]Xd₀d₁d₂d₃
the-1.030.770.672.30
bank0.891.95-0.51-0.24
raised-0.30-0.820.612.30
rates1.47-2.120.890.48
Original research
Follow the math to its sources.
The laboratory separates the original transformer from common modern decoder variants instead of pretending every model uses one architecture.
Architecture · 2017Attention Is All You NeedVaswani et al. ↗Position · 2021RoFormer / rotary position embeddingSu et al. ↗Optimization · 2014Adam: A Method for Stochastic OptimizationKingma & Ba ↗Decoding · 2019The Curious Case of Neural Text DegenerationHoltzman et al. ↗Inference · 2023Grouped-query attentionAinslie et al. ↗Serving · 2023PagedAttention and KV-cache memoryKwon et al. ↗