Linear Regression by Least Squares
Fit a straight relationship by choosing coefficients that minimize the sum of squared residuals. With two features the predictions form a plane. Each coefficient gives predicted change per unit of its feature, holding the others fixed. This alone does not establish cause.
What a learner can do afterwards
- Fit a line by least squares and interpret its slope in the units of the data
- Extend to two features and say what each coefficient means
- Calculate residuals and interpret patterns in a residual plot without treating them as proof of predictive quality
1 · Read
Write a predicted value as y hat. A residual is r = y - y hat: observed minus predicted. It is positive above the fitted line and negative below it. Least squares chooses coefficients that minimize the sum of r^2 over all observations. Squares are nonnegative, so opposite signs cannot cancel. Compare candidate fits on the same data by squaring each residual and adding the squares.
You have used a fitted line to predict. Here you also learn to choose its coefficients. For y hat = a + bx, b is predicted change per extra unit of x. The intercept a is the prediction at x = 0. If predicted score is s hat = 40 + 5h, the slope is 5 points per hour. At h = 4, the prediction is 40 + 5(4) = 60 points. Actual scores may differ. The fitted association alone does not show that studying caused the change.
For ordinary unweighted least squares with an intercept, find mean x and mean y. Subtract these means from each paired x and y to get dx and dy. Let Sxy be the sum of dx*dy and Sxx the sum of dx^2. If x values vary, b = Sxy/Sxx and a = mean y - b*mean x. Substituting mean x into the fitted line gives mean y. A line fitted without an intercept need not pass through this mean point.
Fit (0,1), (1,3), (2,2). The means are x = 1 and y = 2. The dx values are -1, 0, 1; the paired dy values are -1, 1, 0. Products dx*dy total 1, and dx^2 totals 2. Thus b = 1/2 = 0.5 and a = 2 - 0.5(1) = 1.5. The fitted line is y hat = 1.5 + 0.5x.
With two features, y hat = b0 + b1*x1 + b2*x2. Multiply each feature by its coefficient and add b0. Each coefficient gives predicted change per unit of its feature, holding the other fixed. For price in dollars, size in m^2 and age in years, the coefficient units are dollars per m^2 and dollars per year. The signs give the change's direction. Residuals are observed minus predicted. These changes need not be causal.
To fit two features by unweighted least squares with an intercept, use three balances: sum r = 0, sum(x1*r) = 0 and sum(x2*r) = 0. Every sum runs over the training rows. Substitute r = y - b0 - b1*x1 - b2*x2 and collect terms to form linear equations. To solve, subtract multiples of equations to cancel coefficients, then substitute back. The resulting coefficients minimize the squared residual total; individual residuals need not be zero.
Solve all coefficient equations together.
For rows (x1,x2,y) = (0,0,1), (1,0,3), (0,1,4), (1,1,7), the balances give 4b0+2b1+2b2=15, 2b0+2b1+b2=10 and 2b0+b1+2b2=11. Subtract half the first equation from the second to get b1=2.5, and from the third to get b2=3.5. Substitute back to get b0=0.75. The squared residual total is 0.25.
Plot each residual vertically against its fitted value or an input horizontally. Scatter around zero without a clear pattern supports the model, but does not prove that it predicts new data well. A U-shaped residual pattern suggests that the straight model misses curvature. Check predictions on data not used for fitting.
A fan-shaped residual spread suggests that errors have different variability across the range. Their center may still follow a straight relationship. Uncertainty estimates describe how precise predictions or coefficient estimates are. Before relying on estimates that assume constant error spread, investigate the fan and use a method that accounts for changing spread.
Minimize squared residuals, read coefficients as predicted change in stated units, and check residual patterns and predictions on new data.
Lesson, questions and flashcards written for this dataset. Regression references informed the mathematical checks and question design; their prose and code are not reproduced.
Four readings and four additional questions written for this dataset. Textbook and problem-library records were studied for mathematical facts and task design; their wording is not reproduced.
More to read
Compare fits using squared residuals
A prediction and an observation have different jobs. The prediction is the model's output for a given input. The observation is the value recorded in the data. Subtract the prediction from the observation to obtain a residual.
If the model predicts 14 and the recorded value is 11, the residual is -3. Its sign tells you that the prediction is too high. A residual of +3 would mean the prediction is too low by the same amount.
Least squares compares squared residual totals on the same observations. Residuals -3 and +3 each contribute 9. They cannot cancel as they would in a signed sum. A zero residual contributes zero.
Minimizing this total fits the selected model to the training data. It does not establish that the model's shape is suitable or that it will predict later observations accurately. Those questions need separate checks.
Calculate a one-feature fit
For ordinary unweighted least squares with a fitted intercept, start by calculating mean x and mean y. Subtract each mean from the corresponding values. Keep the x and y deviations from each observation paired.
Add the products of paired deviations to obtain Sxy. Add the squared x deviations to obtain Sxx. When the x values vary, Sxx is positive and the slope is Sxy divided by Sxx. Calculate the intercept afterward: mean y minus slope times mean x.
The intercept calculation also explains why this line passes through the mean point. At mean x, the prediction adds back the slope-times-mean-x term that the intercept formula subtracted. The result is mean y.
These conditions matter. A line forced through zero does not have a freely fitted intercept and need not pass through the mean point. If every x value is identical, Sxx is zero and the slope formula cannot be used.
Keep coefficient units attached
Suppose a model predicts price in dollars using floor area in square metres and age in years. An area coefficient of 250 means that increasing area by one square metre adds $250 to the prediction, with age fixed.
An age coefficient of -900 means that one additional year subtracts $900 from the prediction, with area fixed. These are two separate comparisons. Neither statement promises what will happen to the sale price of a particular house.
The coefficients have different units: dollars per square metre and dollars per year. Comparing 250 with 900 does not tell you which feature matters more. Changing area from square metres to another unit would change its coefficient while leaving the same model's predictions unchanged.
A coefficient describes the relationship represented by the fitted equation. An observational association alone does not establish what changing a feature would cause. That requires evidence beyond the coefficient.
Check the equations and the residuals
A two-feature model predicts y hat = b0 + b1*x1 + b2*x2. For unweighted least squares with an intercept, the residual total and the two feature-weighted residual totals are zero. Substitute observed minus predicted for each residual to obtain linear equations for the coefficients.
Solve the equations together. The solution is unique when no input column, including the intercept column of ones, is a linear combination of the others. If columns repeat information exactly, different coefficient values may give the same fitted predictions.
After fitting, plot residuals against an input or the fitted values. A U-shaped pattern suggests that the mean model misses curvature. A fan suggests that error variability changes across the range. The center can follow a straight relationship even when the spread changes.
A plot without a clear pattern supports the model but does not certify its predictions. Check held-out observations as well. Coefficient stability alone also does not establish cause or good future predictions.
2 · Watch
Take it off screen
Where it sits
Learn first
Optional connections
12 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.