Gradient Descent: Following the Slope Downhill
When no formula gives the best parameters, start somewhere and repeatedly step against the slope of the error. The step size decides whether the search creeps, oscillates or diverges, and this one procedure trains almost everything that follows.
What a learner can do afterwards
- Run gradient descent by hand on a one-parameter error function for three steps
- Show what too large and too small a step size each do to the search
- Say why the walk stops at the first flat spot it reaches, usually a local minimum, and what that costs
1 · Read
Last stop you found a minimum exactly by solving f'(x) = 0. Many errors have no formula for their best parameters, so you start somewhere and repeat one move: step against the slope of the error, by an amount set by a step size. When the step size is small enough, no step raises the error, and every step taken off a flat spot lowers it.
The step size rules the search. A tiny size creeps and takes forever. A huge size overshoots, swings back and forth, or diverges. A balanced size walks down steadily. In practice you stop when the slope is nearly zero. Do not judge by the size of the steps alone: each step is the step size times the slope, so a tiny step size makes every step small whatever the slope.
Take error w squared, start at w equal to 4, step size 0.1. The slope is 2w, so the first step is 4 minus 0.8, which lands at 3.2. The next steps give 2.56, then about 2.05. Try step size 1 instead: 4 minus 8 lands at -4, and the walk bounces between 4 and -4 forever. Step size 2 sends it to -12, and the error grows from 16 to 144.
Each step is 0.1 times the slope 2w, so the steps shrink as the walk nears zero.
The walk halts wherever the slope is zero, because a zero slope means a zero step. Usually that is a valley bottom, a local minimum, but not always. Take error w⁴ - 2w² and start at w = 0: the slope 4w³ - 4w is zero, so the walk never moves, though the valleys sit at w = 1 and w = -1. A start just off the top slides away. A slope near zero is all the stopping rule sees, and it cannot tell a valley bottom from a hilltop or a long gentle slope. It cannot promise the global best, which is the price of the method.
Step against the slope with a sane size, stop when the slope is nearly zero, and accept that the flat spot you stop at, usually a valley, may not be the deepest.
2 · Watch
به انگلیسیTake it off screen
Where it sits
Learn first
Optional connections
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.