Multiple Regression and Model Diagnostics
Fit a response using several predictors. Interpret each coefficient as fitted response change with the other predictors held fixed. Check collinearity, influential observations and overfitting when assessing the model.
What a learner can do afterwards
- Calculate fitted values and joint fitted-response changes; interpret a coefficient with the other predictors held fixed
- Identify near and exact collinearity and influential observations, and explain their effects on the fitted model
- Use residual diagnostics and more than training R² to assess model adequacy and compare candidates
1 · Read
The single-predictor model extends to y = a + b1x1 + b2x2 + e. For the usual exact t inference, errors have mean zero given the predictors, constant variance, independence across observations and conditional normality. In an additive fitted model, a coefficient gives the change in fitted response per unit of its predictor, with the others fixed. This describes association; observational regression alone does not establish causation.
For fitted y = 4 + 2x1 - 3x2, insert every predictor value. At x1 = 3 and x2 = 1, fitted y is 4 + 6 - 3 = 7. Holding x2 fixed, a one-unit increase in x1 adds 2 to fitted y. If both predictors change, calculate 2 times the change in x1 minus 3 times the change in x2. The intercept cancels when comparing two fitted values.
Collinearity means predictors carry overlapping linear information. Strong near dependence can inflate standard errors and make coefficients sensitive to small data changes. Predictions near observed predictor combinations may remain stable. With exact dependence, one design-matrix column is a linear combination of others. The matrix loses rank and ordinary least-squares coefficients are not unique. This can happen without large pairwise correlations.
A residual is observed minus fitted y. With an intercept, ordinary least-squares residuals sum to zero, apart from rounding; this does not establish model adequacy. Check residuals for curves and changing spread. An observation is influential if removing it changes the fitted model substantially. Unusual predictor values or a large residual alone do not establish influence. Investigate influential observations; do not delete valid data automatically.
For nested ordinary least-squares fits with an intercept, use the same observations and nonconstant response. Training R² = 1 - SSE/SST, where SST sums squared gaps from mean y. Adding predictors cannot increase SSE: setting their coefficients to zero recovers the old fit. SST stays fixed, so R² cannot fall but may stay equal. Adjusted R² accounts for predictor count and may fall. Model choice also needs residuals, influence, held-out errors and the model's purpose.
Overfitting means learning sample-specific noise that predicts poorly on new data. Extra predictors can raise this risk, but higher training R² alone does not establish overfitting. Check predictions on data not used to fit or select the model. Predict near combinations of predictor values represented in the data; being inside each predictor's separate range is not enough. Even nearby predictions depend on model adequacy and uncertainty.
Read coefficients with other predictors held fixed. Check collinearity, influence and residuals, and compare performance beyond the training fit.
Lesson, questions and flashcards written for this dataset.
Readings and extra questions written for this dataset. Video references list Khan Academy as the publisher; video terms are unknown.
More to read
What stays fixed in a coefficient comparison?
An additive regression combines an intercept with a separate term for each predictor. A coefficient describes how the fitted response changes when its predictor increases by one unit and the remaining predictors stay fixed.
Consider fitted y = 12 + 0.6x1 - 0.4x2. If x1 increases by one while x2 stays fixed, fitted y increases by 0.6. If x2 also increases, its contribution must be included. A coefficient does not describe every comparison in which its predictor happens to increase.
A two-variable correlation and a multiple regression coefficient can give different summaries. Correlation describes linear co-movement without adjustment for other variables. The regression coefficient describes a comparison conditional on the predictors included in that model. Neither naming variables nor holding measured predictors fixed establishes causation. Interpreting an intervention requires a suitable study design and causal assumptions.
Keep every signed contribution
For fitted y = 9 + 3x1 - 2x2, suppose x1 = 2 and x2 = 4. The predictor contributions are 6 and -8, so the fitted response is 9 + 6 - 8 = 7. The negative coefficient belongs to the whole x2 term.
You can also compare fitted values without calculating each one separately. Multiply each predictor's change by its coefficient and add the results. In this equation, increasing x1 by two units and x2 by one unit changes fitted y by 3(2) - 2(1) = 4. The intercept cancels because it is the same in both fitted values.
These are calculations within an additive fitted equation. They do not guarantee that an individual response will change by that amount. Responses vary around fitted predictions, and a poorly supported predictor combination can make the calculation unreliable for prediction.
Different reasons to question a fit
Near collinearity makes the predictors' separate contributions hard to estimate precisely. Coefficients may change substantially after small data changes, even when fitted predictions near observed predictor combinations remain stable. Under the usual mean-zero-error assumptions, near collinearity alone does not make ordinary least-squares estimates biased.
Exact dependence changes what can be identified. When one design column is a linear combination of others, different coefficient sets can produce the same fitted responses. The individual coefficients are not unique.
Influence is a different question: how much does one observation change the fitted model? Compare a fit using all observations with a fit omitting that observation. Unusual predictor values or a large residual can prompt investigation, but neither alone establishes influence. Check possible recording errors and whether the observation belongs in the study. Do not remove valid observations merely because their influence is large.
Training fit and prediction are different checks
For nested ordinary least-squares fits with an intercept, compare the same observations and nonconstant response. Adding predictors cannot lower training R² because the larger model can recover the old fit by setting its new coefficients to zero. The best larger fit may improve on that result or leave it unchanged.
Adjusted R² accounts for predictor count and can decrease. It does not replace residual checks, influence checks or assessment of predictions beyond the training sample. When choosing a model for prediction, compare errors on data not used to fit or select it. A high training R² alone does not establish either good prediction or overfitting.
Also check the proposed predictor combination. Each value can lie inside its separate observed range while the combination remains unlike anything observed. Predictions near observed combinations still carry uncertainty and depend on model adequacy.
Explore more
2 · Watch
Take it off screen
Where it sits
Optional connections
Where this leads
Jobs that lean on this skill. Follow one to see everything it is built on.
12 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.