Gradient Descent: Following the Slope Downhill · seed 1 · A4, ink-friendly. The answer key prints on its own page for grown-ups.

Walking downhill until the slope goes flat

Computing · Machine Learning · ages 19-20
Name ______________________   Date ____________

  1. What does gradient descent do?

    • Solves for the best parameters with one formula
    • Steps against the slope many times
    • Guesses at random until it gets lucky

  2. The slope is still steep, yet each step barely moves. What is wrong?

    • The step size is too small
    • The slope is too gentle to push the walk along
    • The error has no minimum

  3. Error w squared, w is 5. With step size 0.2 the first step lands at 3. Make the step size ten times larger, 2, and give where the first step lands.

    Answer: ______________

  4. The stopping rule ends the walk when the slope is nearly zero. What kind of spot can that be?

    • Only the lowest point of the whole error surface, every single time, whatever the starting point
    • Any nearly flat spot: usually a local minimum, but a flat top or a gentle slope counts too
    • Only a local minimum, never a flat top or a gentle slope

  5. The steps swing wider each round and the error grows. What happened?

    • The size is far too large, so the search diverges
    • The valley is far too smooth, so no size can work
    • The start was far too good

  6. Error w squared, w is 5, step size 0.2. Give w after three steps.

    Answer: ______________

  7. Two runs from different starts can end at different minima.

    Circle one:   True   False

  8. A run explodes, and halving the size still explodes. What should you try next?

    • Double the size back up
    • Freeze all parameters forever and stop searching
    • Keep shrinking the size until it descends
LightMySky · lightmysky.comW1-mt_EjfIT1WgMX-s1

Answer key

For grown-ups. Fold this page away before handing over the rest.

Walking downhill until the slope goes flat W1-mt_EjfIT1WgMX-s1

  1. Steps against the slope many times · It repeats small downhill steps because no formula gives the answer.
  2. The step size is too small · Each step is the step size times the slope, so tiny steps on a steep slope mean the step size is tiny.
  3. -15 · Slope 10 times size 2 is 20, and 5 minus 20 is -15. The error jumps from 25 to 225, so a size this large overshoots and the search diverges.
  4. Any nearly flat spot: usually a local minimum, but a flat top or a gentle slope counts too · A slope near zero is all the stopping rule sees. Usually that is a valley bottom, a local minimum, but the lesson's w⁴ - 2w² walk started at w = 0 sits on a flat top with zero slope, a long gentle slope can trip the rule as well, and nothing promises the deepest valley.
  5. The size is far too large, so the search diverges · Overshooting steps grow instead of settling, which is divergence.
  6. 1.08 · Each step takes away 0.2 times the slope 2w, so w becomes 0.6 times itself: 5 goes to 3, then 1.8, then 1.08.
  7. True · Each run stops at the flat spot its own start leads to, usually a valley bottom, and different starts can lead to different ones.
  8. Keep shrinking the size until it descends · Divergence means the size is still too large, so shrink it further.
Worksheet · LightMySky