Walking downhill to the local bottom

Computing · Machine Learning · ages 19-20

Name ______________________ Date ____________

The steps creep and barely move. What is wrong?
- The step size is too small
- The slope is far too steep
- The error has no minimum
What does gradient descent do?
- Solves for the best parameters with one formula
- Steps against the slope many times
- Guesses at random until it gets lucky
What kind of minimum does the method find?
- Always the global one
- A local one
- None, it never stops
The steps swing wider each round and the error grows. What happened?
- The size is far too large, so the search diverges
- The valley is far too smooth
- The start was far too good
Error w squared, w is 4, step size 0.1. Give w after one step.
Answer: ______________
Each gradient step moves with the slope, uphill.
Circle one: True False
Two runs from different starts can end at different minima.
Circle one: True False
A run explodes, and halving the size still explodes. What should you try next?
- Double the size back up
- Freeze all parameters forever
- Keep shrinking the size until it descends

Answer key

For grown-ups. Fold this page away before handing over the rest.

The step size is too small · A tiny size means tiny progress on every round.
Steps against the slope many times · It repeats small downhill steps because no formula gives the answer.
A local one · It settles into the valley it walked into, which may not be the deepest.
The size is far too large, so the search diverges · Overshooting steps grow instead of settling, which is divergence.
3.2 · Slope 8 times size 0.1 is 0.8, and 4 minus 0.8 is 3.2.
False · It moves against the slope, downhill toward lower error.
True · Each run falls into its own nearest valley.
Keep shrinking the size until it descends · Divergence means the size is still too large, so shrink it further.