Overfitting and the Bias-Variance Trade-off
Overfitting can occur when a model follows training-sample noise and predicts poorly on new data. Bias is the difference between its average prediction across training samples and the true mean response. Variance measures prediction changes across samples. For squared prediction error, expected error includes squared bias, variance and irreducible noise.
What a learner can do afterwards
- Interpret training and held-out error patterns as flexibility grows, and select a model using the lowest held-out validation error
- Diagnose bias or variance from stated symptoms, explain their coexistence, and calculate prediction variance from equally likely training samples
- Explain how more representative training data can reduce variance without removing model bias
1 · Read
At a fixed input, bias is the average prediction over training samples minus the true mean response. A model that always predicts 0.3 when the true mean is 0.5 has bias -0.2. A straight line can have high bias when the pattern curves. More data alone cannot make that line bend.
At a fixed input, variance measures how much predictions differ across training samples. Compute it by averaging their squared distances from their average. If equally likely samples give predictions 1 and 5, their average is 3. The squared distances are 4 and 4, so the variance is 4. A flexible model can have high variance when it follows sample noise, but flexibility does not guarantee it.
When more flexible models include the simpler models and fitting minimises the same loss, training error cannot increase. Held-out validation data is not used to fit. Its error can fall, then rise as flexibility grows. Falling training error with rising held-out error suggests overfitting. Compare models on the same held-out sample and choose the lowest held-out error, rather than the lowest training error.
Bagging means bootstrap aggregating. Fit several models to bootstrap samples, drawn with replacement from the training data, then average their predictions. Averaging can reduce variance when the models do not make identical sampling errors. It does not guarantee that all bias or noise disappears.
Bias is the average prediction minus the true mean response. An average below the truth means negative bias. Variance measures spread across training samples. A smaller spread means lower variance, even if bias remains. A model can have both. More representative data from the same population often reduces variance, but need not remove model bias. A straight line still cannot represent a curved pattern.
Video correction: the publisher reports an error in the variance definition at 4:06 of `EuBBz3bI-aA`. Use this definition: at a fixed input, prediction variance is the average squared distance of predictions from their average across training samples. High variance describes a wide spread across samples. It does not guarantee a large change after every small edit to one sample.
Bias is an average prediction error; variance is variation across training samples. Use held-out performance to choose flexibility. More data or bagging can reduce variance.
The reading and all three extra questions were independently written for this dataset. The reading explains bias, variance, overfitting and bagging using established statistical concepts. No imported reading, media or question remains in this file.
2 · Watch
Take it off screen
Where it sits
Optional connections
11 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.