Cross-Validation and Honest Model Selection
Every choice made by looking at a set of data uses that set up, so the score on it stops being an estimate of future performance. Splitting into folds, and holding a final set back untouched, is how a reported number stays believable.
What a learner can do afterwards
- Run k-fold cross-validation to choose a penalty strength
- Explain what leaks when the test set is used to pick anything
- Say why a time-ordered dataset must not be split at random
1 · Read
Every choice made by looking at a set of data uses that set up, so the score on it stops being an estimate of future performance. That is why honest analysts use cross-validation: split the data into folds, train on some, check on others, and repeat so every row gets a fair turn as the check. The average score tells you how the model behaves on data it has not memorised.
Take 1000 rows and 5 folds. The rows split evenly, so each validation fold holds 1000 divided by 5, which is 200 rows. Each fold takes a turn as the check while the other four train, so the model is trained from scratch 5 times. To choose a penalty strength, pick the one with the best average score across the validation folds.
There is a final rule that keeps everything honest: hold one set of data back completely. Do not touch it while choosing settings or comparing models. Score it once, at the very end. If you keep testing candidates on it while deciding, you quietly teach your choices to please that set, and its number stops telling the truth about the future.
When rows have no time order, assign each row to a fold at random rather than by position in the file, so every fold stays representative. But a time-ordered dataset must never be split at random: train on the older rows and validate on the rows that come after them. Random splits would put future rows into training, letting the model cheat by peeking ahead.
Choose with folds and averages, judge once on untouched data, and keep time order on the correct side of the split.
2 · Watch
Take it off screen
Where it sits
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.