Tool 05
Tune architecture knobs and trace how data flows through each transformer block.
A transformer stacks repeated blocks of attention and feed-forward layers. Each block refines the representation. Depth (num_layers) lets the model build increasingly abstract patterns — from local grammar to broader context. This tool lets you tune the knobs and trace the data flow.
Each transformer block contains:
1. Multi-head self-attention (tokens look at each other) 2. A feed-forward network (per-token MLP) 3. Residual connections (add input back to output) 4. Layer normalization (stabilize training)
Residual connections let gradients flow through deep networks — without them, training 96+ layers would be nearly impossible.
Instead of one attention pass, the model splits d_model into multiple "heads" — each head learns different relationship patterns. One head might track syntax, another coreference, another local bigrams.
Head outputs are concatenated and projected back to d_model. num_heads must divide d_model evenly.
The same block repeats num_layers times. Early layers often capture syntax and local structure. Deeper layers capture more abstract, task-relevant patterns.
Data flows: embed → block₁ → block₂ → ... → blockₙ → output head. Each step transforms shape [seq_len, d_model] while keeping that shape through the stack.
Parameters grow roughly with:
• d_model² (attention projections and FFN layers) • num_layers (stacking multiplies everything) • d_ff (feed-forward inner dimension, usually 4× d_model)
Crank num_layers from 1 to 8 in the playground above and watch the parameter estimate jump. GPT-3 175B has 96 layers and d_model=12288 — now you know where the billions come from.
Start with defaults and read each step's shape annotation. Increase seq_len and see how attention cost grows quadratically (seq_len²).
Try d_model=8 with num_heads=2, then num_heads=4 — notice d_model must be divisible by num_heads. Push num_layers up and connect parameter count to model file sizes in the GGUF inspector.