Fitting a Model to Data: Least Squares, Chi-Square and Goodness of Fit
Fit models by least squares, using uncertainty weights when appropriate. Report parameter uncertainties under stated error assumptions. Use residuals and chi-square to assess consistency with those assumptions without treating a good fit as proof of the model.
What a learner can do afterwards
- Fits a straight line by weighted least squares and reports parameter uncertainties
- Interprets a reduced chi-square value, including what too small a value suggests
- Reads structure in the residuals and says what it implies about the model
1 · Read
A residual is observed y minus predicted y-hat at the same x. Ordinary least squares chooses parameters that minimise the sum of squared residuals, called SSE. Squaring prevents opposite signs from cancelling. A fit is possible without reported point uncertainties. Its standard errors and statistical interpretation need justified assumptions about the measurement errors.
For a constant model y-hat = a with equal weights, the least-squares estimate is the arithmetic mean. Add the readings and divide by their number. Readings 1, 4 and 7 give a = 12/3 = 4, with residuals -3, 0 and 3. The mean need not equal the middle reading.
Treat x as exact here. For independent measurement errors with known standard deviations sigma, weighted least squares minimises chi-square: add (residual/sigma)² over all points. Each squared residual has weight 1/sigma². Smaller sigma gives a larger penalty for the same residual. Precision does not rule out bias. Residual and sigma use the same units, making their ratio dimensionless.
For residuals -3, 0 and 3 with sigma = 2 for every reading, first divide by sigma. The ratios are -1.5, 0 and 1.5. Squaring gives 2.25, 0 and 2.25. Add them to get chi-square = 4.5.
A line predicts y-hat = a + b x. Intercept a has y units; slope b has y units per x unit. As in dimensional analysis, check the units of each term. Set w = 1/sigma² and W = sum(w). The weighted means are x-bar = sum(w x)/W and y-bar = sum(w y)/W. Here sum means add over all points.
Calculate Cxx = sum[w(x - x-bar)²] and Cxy = sum[w(x - x-bar)(y - y-bar)]. If Cxx > 0, the weighted line has b = Cxy/Cxx and a = y-bar - b x-bar. It passes through the weighted mean point. If every x is equal, Cxx = 0 and separate slope and intercept cannot be determined.
For this line, SE(b) = 1/sqrt(Cxx) and SE(a) = sqrt(1/W + x-bar²/Cxx). sqrt means positive square root. These standard errors assume a correct linear mean, independent errors, correct known variances and exact x. Normal errors support normal confidence intervals. A standard error is not a guaranteed bound. Report each coefficient and its standard error with units.
If every known sigma doubles, every weight becomes one quarter as large. Relative weights stay unchanged, so the fitted line stays unchanged. W, Cxx and Cxy become one quarter as large. Both coefficient standard errors double, while chi-square becomes one quarter as large.
For a full-rank linear model, n measurements and p fitted coefficients give residual degrees of freedom nu = n - p. Full rank means the coefficients can be separately determined. Reduced chi-square is chi-square/nu, requiring nu > 0. With the correct mean and independent normal errors of known correct sigmas, chi-square has mean nu. Reduced chi-square has mean 1 and standard deviation sqrt(2/nu). Near 1 is consistent with these assumptions, not proof.
An unusually small reduced chi-square can suggest overstated uncertainties, ignored correlations or data-selection effects. Chance variation matters, especially with few degrees of freedom. Extra coefficients alone do not explain a small value: nu already accounts for fitted coefficients in the stated linear model. Selecting models after seeing the data can change this interpretation.
Plot residuals against x and measurement order. Repeated curvature suggests missing model structure or another systematic effect. Check the model and measurement process before choosing a remedy. Chance can produce apparent patterns, and featureless scatter does not prove correctness. Fitted residuals can be correlated even when the original errors are independent.
Check residual spread too. Raw residuals may spread out where sigma is larger, even with a suitable model. Divide each residual by its sigma before comparing spread across x ranges. A remaining widening pattern suggests checking the variance assumptions. This scaling does not remove every effect of fitting.
With n > 2 and at least two distinct exact x values, fit an unweighted line. Assume a correct linear mean and independent errors of equal positive variance. Estimate s² = SSE/(n - 2). Evaluate the standard-error formulas with w = 1, then multiply the results by s. Normal errors support t intervals when s > 0. Using s as every sigma gives reduced chi-square 1 only when SSE > 0. If SSE = 0, s = 0 cannot be a chi-square denominator.
Investigate unusually large residuals before removing observations. Check records, instruments and measurement conditions. A valid unusual observation may reveal a model limitation. Document any correction or exclusion. Comparing fits with and without a questionable observation can reveal sensitivity; report why you made that comparison.
State the error assumptions, fit the coefficients, report their standard errors, and check residuals and chi-square without treating consistency as proof.
Lesson, questions and flashcards written for this dataset.
Additional questions, readings and flashcards written for this dataset.
More to read
What a fitted line establishes
Choose a line model y-hat = a + b x. Each observed y has a prediction at the same x, and their signed difference is its residual. Ordinary least squares minimises the total squared residual. Specified weights change how much each squared residual contributes.
The resulting coefficients solve that fitting problem. This does not make the line a physical law or establish how well it predicts new observations. Those conclusions need further evidence.
State the fitted equation, coefficient units and standard errors. Explain the assumptions used to calculate those errors. A standard error is not a promise that the true coefficient lies within that distance.
Use particular care outside the measured x range. A line can fit the observed range while missing a change beyond it.
Different chi-square statistics
For measurements with known independent normal errors, add squared ratios of residual to measurement standard deviation. In a full-rank linear fit with n measurements and p fitted coefficients, the reference chi-square distribution has n - p degrees of freedom when the mean model and variances are correct.
Categorical counts use a different construction: add (observed count - expected count)²/expected count. With k categories, a fixed total and specified category probabilities, the usual degrees of freedom are k - 1. Estimating probabilities from those counts can reduce the degrees of freedom. Small expected counts can make the reference approximation unsuitable.
Identify which sampling model applies before choosing a formula or reference distribution. A statistical test also needs a stated decision rule. Not rejecting a model does not establish its truth.
Comparing fits on the same scale
A comparison needs a clear target. State the candidate models, observations and assumptions about measurement error. Check whether each model leaves patterns in its residuals.
Adding adjustable coefficients can lower a training-data residual sum. That alone does not establish better prediction. Check coefficient uncertainty and, where possible, prediction on observations that were not used to fit or choose the model.
A transformation can also change the target. Minimising squared gaps in log(y) is a different objective from minimising squared gaps in y. The two residual sums are on different scales and express different assumptions about error. Do not compare their numerical sizes as though they measure the same discrepancy.
Describe what the comparison supports and where the evidence remains limited.
Checking an unusual observation
Start with the measurement record when an observation looks unusual. Check its units, transcription, instrument settings and experimental conditions. Correct an established error when the correct information is available.
A large residual measures disagreement with the fitted prediction. An unusual x value is a different property: it can give an observation more potential to change the fitted line. An observation's actual influence also depends on its residual and the other data.
A valid observation can expose a limitation in the model. Removing it only because the fit improves would hide that evidence. A comparison with and without the observation can show sensitivity, but report why you made the comparison. Document the rule and evidence behind any exclusion.
What R-squared measures
Let TSS be the sum of squared differences between observed y values and their sample mean. Let SSE be the fitted model's sum of squared residuals. When TSS is positive, R-squared = 1 - SSE/TSS.
For an ordinary least-squares fit with an intercept, this training-data score lies between 0 and 1. It describes the reduction in squared error relative to predicting the sample mean. It is not the percentage reduction in a typical absolute miss. In simple linear regression with an intercept, it equals the squared sample correlation when that correlation is defined.
These conditions matter. With the same score formula, a model without an intercept or predictions evaluated on new data can give a negative value. A high training score does not establish causation, correct error assumptions or reliable future predictions. Examine residuals and the intended prediction task too.
2 · Watch
به انگلیسیTake it off screen
Where it sits
Learn first
Optional connections
Where this leads
18 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.