The Transformer Block: Heads, Residuals and Normalisation
One block stacks multi-head attention and a position-wise network around a residual path, with normalisation holding the scale steady. Attention treats a sequence as a set, so position has to be supplied explicitly or an unmasked block cannot tell an order from a shuffle.
What a learner can do afterwards
- Trace a single vector through one block and name what each sub-layer changed about it
- Say why several narrow heads are used instead of one wide one
- Predict what fails if the residual path is cut, and what fails if position is never encoded
1 · Read
Attention treats a sequence as a set, so position must be supplied explicitly. Without position codes, an unmasked block is order-blind: shuffle the inputs and the outputs shuffle the same way, with nothing else changed. A causal mask, which lets each position see itself and the positions before it but nothing after, breaks that symmetry, so some order gets through even without codes, and whether the final answer moves depends on how the outputs are read out.
Follow one vector through the block. Attention mixes in the other positions, the residual adds the input back, normalisation steadies the scale, and the position-wise network transforms each position alone. Residual plus norm closes each half of the block.
Each piece earns its place. Cut the residual path and deep stacks become very hard to train, because the signal and its gradient degrade layer by layer. Several narrow heads are used instead of one wide head because each head can watch a different relation within the same budget.
For an unmasked block, run the shuffle test when outputs look order-blind. If shuffling the input only shuffles the output, position was never encoded or the codes are being ignored.
Position codes supply order, residuals carry the signal deep, and narrow heads divide the watching.
2 · Watch
Take it off screen
Where it sits
Learn first
Optional connections
This opens up
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.