--- title: "Cross-Validation and Honest Model Selection" description: "Every choice made by looking at a set of data uses that set up, so the score on it stops being an estimate of future performance. Splitting into folds, and holding a final set back untouched, is how a" canonical: https://lightmysky.com/learn/computing/cross-validation-and-honest-model-selection-mt_YG7X3HYCrT source: https://lightmysky.com/learn/computing/cross-validation-and-honest-model-selection-mt_YG7X3HYCrT.md retrieved: 2026-09-12 --- > **Agent view.** This is the Markdown twin of the page, for tools and assistants. > When to use this site, and the call that answers each job: https://lightmysky.com/agent-instructions.md > API description (OpenAPI 3.1): https://lightmysky.com/openapi.json · Authentication: https://lightmysky.com/auth.md > Pricing: https://lightmysky.com/pricing.md · Catalog: https://lightmysky.com/llms.txt · Full catalog: https://lightmysky.com/llms-full.txt > Every machine-readable file on this domain: https://lightmysky.com/.well-known/ai-catalog.json > Ask for Markdown with `Accept: text/markdown`, a `.md` address, or `?mode=agent`. # Cross-Validation and Honest Model Selection Every choice made by looking at a set of data uses that set up, so the score on it stops being an estimate of future performance. Splitting into folds, and holding a final set back untouched, is how a reported number stays believable. Subject: Computing · Area: Machine Learning · Ages 20 to 21 Page: https://lightmysky.com/learn/computing/cross-validation-and-honest-model-selection-mt_YG7X3HYCrT ## Ready when they can - Run k-fold cross-validation to choose a penalty strength - Explain what leaks when the test set is used to pick anything - Say why a time-ordered dataset must not be split at random ## Lesson: Folds for choosing, one vault for judging Every choice made by looking at a set of data uses that set up, so the score on it stops being an estimate of future performance. That is why honest analysts use cross-validation: split the data into folds, train on some, check on others, and repeat so every row gets a fair turn as the check. The average score tells you how the model behaves on data it has not memorised. **Example.** Take 1000 rows and 5 folds. The rows split evenly, so each validation fold holds 1000 divided by 5, which is 200 rows. Each fold takes a turn as the check while the other four train, so the model is trained from scratch 5 times. To choose a penalty strength, pick the one with the best average score across the validation folds. There is a final rule that keeps everything honest: hold one set of data back completely. Do not touch it while choosing settings or comparing models. Score it once, at the very end. If you keep testing candidates on it while deciding, you quietly teach your choices to please that set, and its number stops telling the truth about the future. **Tip.** When rows have no time order, assign each row to a fold at random rather than by position in the file, so every fold stays representative. But a time-ordered dataset must never be split at random: train on the older rows and validate on the rows that come after them. Random splits would put future rows into training, letting the model cheat by peeking ahead. **Recap.** Choose with folds and averages, judge once on untouched data, and keep time order on the correct side of the split. ## Practice 16 questions on this page, each with its working shown. ## Needs first - [Sampling Methods and Their Trade-offs](https://lightmysky.com/learn/mathematics/sampling-methods-and-their-trade-offs-mt_aSsRQvnKa6) - [Regularisation: Charging for Complexity](https://lightmysky.com/learn/computing/regularisation-charging-for-complexity-mt_RVgsDP0XzP) ## Opens up - [Beyond Accuracy: Precision, Recall and the Cost of an Error](https://lightmysky.com/learn/computing/beyond-accuracy-precision-recall-and-the-cost-of-an-error-mt_BHGgZM7tak) - [Benchmarks and the Traps in Them](https://lightmysky.com/learn/computing/benchmarks-and-the-traps-in-them-mt_o9zLZbkaen) - [Scaling Laws and the Compute Budget](https://lightmysky.com/learn/computing/scaling-laws-and-the-compute-budget-mt_tzl0sQblIM)