Overfitting and the Bias-Variance Trade-off
A model flexible enough to fit the noise will do it, and then fail on anything new. Error splits into bias from a model too simple to represent the pattern and variance from a model too sensitive to the particular sample it saw.
What a learner can do afterwards
- Show training error falling while held-out error rises as flexibility grows
- Attribute a stated failure to bias or to variance and say why
- Explain why more training data helps variance more than it helps bias
1 · Read
Bias is error from a model too simple or rigid to catch the pattern. Picture fitting a straight line to data that curves: it misses the same way every time, and bias is the average prediction minus the true value, so a constant 0.4 against truth 0.5 means bias minus 0.1.
Variance is the opposite trouble: the model is so flexible it hugs every wobble of its sample, fitting noise instead of pattern. It shines on training data yet fails on new data, and it swings a lot whenever the sample changes.
Watch flexibility grow: training error keeps falling while held out error first falls then rises, and the rising tail is overfitting. The middle ground catches the pattern but ignores noise, and averaging many models trained on resampled data cuts variance.
Name each failure before fixing it: steady same way misses mean bias, big shifts with new samples mean variance. More training data calms variance far more than bias, since no pile of data straightens a model too stiff to bend.
Bias misses steadily from stiffness, variance swings from clinging to noise, and good models sit between.
2 · Watch
Take it off screen
Where it sits
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.