--- title: "Linear Regression by Least Squares" description: "Fit a straight relationship by choosing coefficients that minimize the sum of squared residuals. With two features the predictions form a plane. Each coefficient gives predicted change per unit of its" canonical: https://lightmysky.com/learn/computing/linear-regression-by-least-squares-mt_QRRkpIrwWs source: https://lightmysky.com/learn/computing/linear-regression-by-least-squares-mt_QRRkpIrwWs.md retrieved: 2026-11-12 --- > **Agent view.** This is the Markdown twin of the page, for tools and assistants. > When to use this site, and the call that answers each job: https://lightmysky.com/agent-instructions.md > API description (OpenAPI 3.1): https://lightmysky.com/openapi.json · Authentication: https://lightmysky.com/auth.md > Pricing: https://lightmysky.com/pricing.md · Catalog: https://lightmysky.com/llms.txt · Full catalog: https://lightmysky.com/llms-full.txt > Every machine-readable file on this domain: https://lightmysky.com/.well-known/ai-catalog.json > Ask for Markdown with `Accept: text/markdown`, a `.md` address, or `?mode=agent`. # Linear Regression by Least Squares Fit a straight relationship by choosing coefficients that minimize the sum of squared residuals. With two features the predictions form a plane. Each coefficient gives predicted change per unit of its feature, holding the others fixed. This alone does not establish cause. Subject: Computing · Area: Machine Learning · Ages 19 to 20 Page: https://lightmysky.com/learn/computing/linear-regression-by-least-squares-mt_QRRkpIrwWs ## Ready when they can - Fit a line by least squares and interpret its slope in the units of the data - Extend to two features and say what each coefficient means - Calculate residuals and interpret patterns in a residual plot without treating them as proof of predictive quality ## Lesson: Drawing the line that misses least Write a predicted value as y hat. A residual is r = y - y hat: observed minus predicted. It is positive above the fitted line and negative below it. Least squares chooses coefficients that minimize the sum of r^2 over all observations. Squares are nonnegative, so opposite signs cannot cancel. Compare candidate fits on the same data by squaring each residual and adding the squares. **Example.** You have used a fitted line to predict. Here you also learn to choose its coefficients. For y hat = a + bx, b is predicted change per extra unit of x. The intercept a is the prediction at x = 0. If predicted score is s hat = 40 + 5h, the slope is 5 points per hour. At h = 4, the prediction is 40 + 5(4) = 60 points. Actual scores may differ. The fitted association alone does not show that studying caused the change. *(drawing: Observed 11 minus predicted 14 gives residual -3. This number line shows subtraction, not a scatter plot.)* For ordinary unweighted least squares with an intercept, find mean x and mean y. Subtract these means from each paired x and y to get dx and dy. Let Sxy be the sum of dx*dy and Sxx the sum of dx^2. If x values vary, b = Sxy/Sxx and a = mean y - b*mean x. Substituting mean x into the fitted line gives mean y. A line fitted without an intercept need not pass through this mean point. **Example.** Fit (0,1), (1,3), (2,2). The means are x = 1 and y = 2. The dx values are -1, 0, 1; the paired dy values are -1, 1, 0. Products dx*dy total 1, and dx^2 totals 2. Thus b = 1/2 = 0.5 and a = 2 - 0.5(1) = 1.5. The fitted line is y hat = 1.5 + 0.5x. *(drawing: Keep each x deviation paired with the y deviation from the same observation.)* With two features, y hat = b0 + b1*x1 + b2*x2. Multiply each feature by its coefficient and add b0. Each coefficient gives predicted change per unit of its feature, holding the other fixed. For price in dollars, size in m^2 and age in years, the coefficient units are dollars per m^2 and dollars per year. The signs give the change's direction. Residuals are observed minus predicted. These changes need not be causal. To fit two features by unweighted least squares with an intercept, use three balances: sum r = 0, sum(x1*r) = 0 and sum(x2*r) = 0. Every sum runs over the training rows. Substitute r = y - b0 - b1*x1 - b2*x2 and collect terms to form linear equations. To solve, subtract multiples of equations to cancel coefficients, then substitute back. The resulting coefficients minimize the squared residual total; individual residuals need not be zero. **Example.** For rows (x1,x2,y) = (0,0,1), (1,0,3), (0,1,4), (1,1,7), the balances give 4b0+2b1+2b2=15, 2b0+2b1+b2=10 and 2b0+b1+2b2=11. Subtract half the first equation from the second to get b1=2.5, and from the third to get b2=3.5. Substitute back to get b0=0.75. The squared residual total is 0.25. *(drawing: The fitted equation is y hat = 0.75 + 2.5*x1 + 3.5*x2.)* Plot each residual vertically against its fitted value or an input horizontally. Scatter around zero without a clear pattern supports the model, but does not prove that it predicts new data well. A U-shaped residual pattern suggests that the straight model misses curvature. Check predictions on data not used for fitting. A fan-shaped residual spread suggests that errors have different variability across the range. Their center may still follow a straight relationship. Uncertainty estimates describe how precise predictions or coefficient estimates are. Before relying on estimates that assume constant error spread, investigate the fan and use a method that accounts for changing spread. **Recap.** Minimize squared residuals, read coefficients as predicted change in stated units, and check residual patterns and predictions on new data. *Lesson, questions and flashcards written for this dataset. Regression references informed the mathematical checks and question design; their prose and code are not reproduced.* ## Practice 12 questions on this page, each with its working shown. ## Needs first - [Data as a Matrix: Rows, Features and the Target](https://lightmysky.com/learn/computing/data-as-a-matrix-rows-features-and-the-target-mt_KEN-TU1Gpn) - [The Regression Line and Making Predictions](https://lightmysky.com/learn/mathematics/the-regression-line-and-making-predictions-mt_tsjctRmkyE) ## Optional connections - [Gaussian elimination and row echelon form](https://lightmysky.com/learn/mathematics/gaussian-elimination-and-row-echelon-form-mt_PZBce3UvRm) - [Correlation and the Product-Moment Coefficient](https://lightmysky.com/learn/mathematics/correlation-and-the-product-moment-coefficient-mt_Rs_Z0P7ELJ) ## Opens up - [Gradient Descent: Following the Slope Downhill](https://lightmysky.com/learn/computing/gradient-descent-following-the-slope-downhill-mt_EjfIT1WgMX) - [Logistic Regression and the Decision Boundary](https://lightmysky.com/learn/computing/logistic-regression-and-the-decision-boundary-mt_FaVIaT7xl2)