--- title: "Gradient Descent: Following the Slope Downhill" description: "When no formula gives the best parameters, start somewhere and repeatedly step against the slope of the error. The step size decides whether the search creeps, oscillates or diverges, and this one pro" canonical: https://lightmysky.com/learn/computing/gradient-descent-following-the-slope-downhill-mt_EjfIT1WgMX source: https://lightmysky.com/learn/computing/gradient-descent-following-the-slope-downhill-mt_EjfIT1WgMX.md retrieved: 2026-11-12 --- > **Agent view.** This is the Markdown twin of the page, for tools and assistants. > When to use this site, and the call that answers each job: https://lightmysky.com/agent-instructions.md > API description (OpenAPI 3.1): https://lightmysky.com/openapi.json · Authentication: https://lightmysky.com/auth.md > Pricing: https://lightmysky.com/pricing.md · Catalog: https://lightmysky.com/llms.txt · Full catalog: https://lightmysky.com/llms-full.txt > Every machine-readable file on this domain: https://lightmysky.com/.well-known/ai-catalog.json > Ask for Markdown with `Accept: text/markdown`, a `.md` address, or `?mode=agent`. # Gradient Descent: Following the Slope Downhill When no formula gives the best parameters, start somewhere and repeatedly step against the slope of the error. The step size decides whether the search creeps, oscillates or diverges, and this one procedure trains almost everything that follows. Subject: Computing · Area: Machine Learning · Ages 19 to 20 Page: https://lightmysky.com/learn/computing/gradient-descent-following-the-slope-downhill-mt_EjfIT1WgMX ## Ready when they can - Run gradient descent by hand on a one-parameter error function for three steps - Show what too large and too small a step size each do to the search - Say why the walk stops at the first flat spot it reaches, usually a local minimum, and what that costs ## Lesson: Walking downhill until the slope goes flat Last stop you found a minimum exactly by solving f'(x) = 0. Many errors have no formula for their best parameters, so you start somewhere and repeat one move: step against the slope of the error, by an amount set by a step size. When the step size is small enough, no step raises the error, and every step taken off a flat spot lowers it. The step size rules the search. A tiny size creeps and takes forever. A huge size overshoots, swings back and forth, or diverges. A balanced size walks down steadily. In practice you stop when the slope is nearly zero. Do not judge by the size of the steps alone: each step is the step size times the slope, so a tiny step size makes every step small whatever the slope. **Example.** Take error w squared, start at w equal to 4, step size 0.1. The slope is 2w, so the first step is 4 minus 0.8, which lands at 3.2. The next steps give 2.56, then about 2.05. Try step size 1 instead: 4 minus 8 lands at -4, and the walk bounces between 4 and -4 forever. Step size 2 sends it to -12, and the error grows from 16 to 144. The walk halts wherever the slope is zero, because a zero slope means a zero step. Usually that is a valley bottom, a local minimum, but not always. Take error w⁴ - 2w² and start at w = 0: the slope 4w³ - 4w is zero, so the walk never moves, though the valleys sit at w = 1 and w = -1. A start just off the top slides away. A slope near zero is all the stopping rule sees, and it cannot tell a valley bottom from a hilltop or a long gentle slope. It cannot promise the global best, which is the price of the method. **Recap.** Step against the slope with a sane size, stop when the slope is nearly zero, and accept that the flat spot you stop at, usually a valley, may not be the deepest. ## Practice 8 questions on this page, each with its working shown. ## Needs first - [Stationary Points and the Second Derivative](https://lightmysky.com/learn/mathematics/stationary-points-and-the-second-derivative-mt_VbGJEFFgfs) ## Optional connections - [Optimisation with Calculus](https://lightmysky.com/learn/mathematics/optimisation-with-calculus-mt_9I0ql39Lzw) - [Directional Derivatives and the Gradient](https://lightmysky.com/learn/mathematics/directional-derivatives-and-the-gradient-mt_LW_KckY5Ad) - [Linear Regression by Least Squares](https://lightmysky.com/learn/computing/linear-regression-by-least-squares-mt_QRRkpIrwWs) - [The Derivative as a Rate of Change](https://lightmysky.com/learn/mathematics/the-derivative-as-a-rate-of-change-mt_RqyLE3jrAW) ## Opens up - [Backpropagation: Assigning Blame for an Error](https://lightmysky.com/learn/computing/backpropagation-assigning-blame-for-an-error-mt_5bwaSWfF96) - [Logistic Regression and the Decision Boundary](https://lightmysky.com/learn/computing/logistic-regression-and-the-decision-boundary-mt_FaVIaT7xl2)