Gradient Descent: Following the Slope Downhill
When no formula gives the best parameters, start somewhere and repeatedly step against the slope of the error. The step size decides whether the search creeps, oscillates or diverges, and this one procedure trains almost everything that follows.
What a learner can do afterwards
- Run gradient descent by hand on a one-parameter error function for three steps
- Show what too large and too small a step size each do to the search
- Say why the method finds a local minimum and what that costs
1 · Read
Some errors have no formula for the best parameters. You start somewhere and repeat one move: step against the slope of the error. Each step lowers the error a little.
The step size rules the search. A tiny size creeps and takes forever. A huge size overshoots, swings back and forth, or diverges. A balanced size walks down steadily.
Take error w squared, start at w equal to 4, step size 0.1. The slope is 2w, so the first step is 4 minus 0.8, which lands at 3.2. The next steps give 2.56, then about 2.05.
The method stops at a local minimum, the bottom of the valley it walked into. It cannot promise the global best, and that is the price of the method.
Step against the slope with a sane size, and accept the local bottom you reach.
2 · Watch
Take it off screen
Where it sits
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.