Backpropagation: Assigning Blame for an Error
Training a network means knowing how much each weight contributed to the error, which the chain rule gives by passing derivatives backwards through the layers. Every weight is then nudged by gradient descent, and the whole procedure is one pass forward and one pass back.
What a learner can do afterwards
- Propagate an error backwards through a two-layer network by hand
- Say what the chain rule contributes at each layer boundary
- Explain why a squashing function with a flat tail slows learning
1 · Read
Training a stack that can fence the opposite yes corners runs one pass forward and one pass back. Forward, inputs flow layer by layer into a prediction. Then a loss function scores the gap between the prediction and the target.
Take y as the square of two x plus one, at x equals one. The inner layer gives three with a local rate of two. The outer square gives a local rate of six, and six times two makes twelve. A second stack, the square of x squared plus one at one, combines four times two into eight.
The backward pass walks the error from the output toward the input. At each layer boundary the chain rule multiplies the local rates together. Static backprop changes only weights and biases, never the wiring. Each weight then takes a small gradient descent nudge downhill against the loss.
A squashing function with a flat tail has a tiny local rate out there. Multiplying by near zero shrinks the signal each layer back, so early layers learn very slowly.
Push forward to score the error, multiply local rates backward, and nudge each weight downhill against the loss.
2 · Watch
Take it off screen
Where it sits
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.