The Transformer Block: Heads, Residuals and Normalisation
One block stacks multi-head attention and a position-wise network around a residual path, with normalisation holding the scale steady. Attention treats a sequence as a set, so position has to be supplied explicitly or the block cannot tell an order from a shuffle.
What a learner can do afterwards
- Trace a single vector through one block and name what each sub-layer changed about it
- Say why several narrow heads are used instead of one wide one
- Predict what fails if the residual path is cut, and what fails if position is never encoded
1 · Read
Attention treats a sequence as a set, so position must be supplied explicitly. Without position codes the block cannot tell an order from a shuffle, and swapping two words would barely change its answer.
Follow one vector through the block. Attention mixes in the other positions, the residual adds the input back, normalisation steadies the scale, and the position-wise network transforms each position alone. Residual plus norm closes each half of the block.
Each piece earns its place. Cut the residual path and deep stacks stop training, because the signal and its gradient degrade. Several narrow heads beat one wide head because each head can watch a different relation within the same budget.
Run the shuffle test when outputs look order-blind. If shuffling the input barely moves the output, position was never encoded.
Position codes supply order, residuals carry the signal deep, and narrow heads divide the watching.
2 · Watch
Take it off screen
Where it sits
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.