Cross-Validation and Honest Model Selection · seed 1 · A4, ink-friendly. The answer key prints on its own page for grown-ups.
En anglais

Folds for choosing, one vault for judging

Computing · Machine Learning · ages 20-21
Name ______________________   Date ____________

  1. You run 4-fold cross-validation. How many separate trainings happen, and what does each one use?

    • 1 training on all 4 folds at once
    • 4 trainings, each on 3 folds with the fourth fold as the check
    • 4 trainings, each on all 4 folds at once with no fold held back
    • 3 trainings, each on 1 fold with the other 3 as the check

  2. You run 5-fold cross-validation. How many times is a model trained from scratch?

    • 1 time
    • 10 times
    • 5 times

  3. After cross-validation picks your model and settings, you should score the final untouched set once and report that number.

    Circle one:   True   False

  4. A dataset has 1000 rows. In 5-fold cross-validation, how many rows are in each validation fold?

    Answer: ______________

  5. A file's rows are sorted by customer region, so the rows of each of its several regions sit together, and there is no time order. Ravi puts the first quarter of the file in fold 1, the next quarter in fold 2, and so on. Each fold will still be representative of the whole dataset.

    Circle one:   True   False

  6. Monthly sales run from January 2019 to December 2024, and you want an honest estimate for 2025. Which split fits?

    • Train on 2019 to 2023 and validate on 2024
    • Shuffle every month into folds at random
    • Train on 2024 alone and validate on 2019 to 2023
    • Validate on every other month across the six years

  7. Three penalty strengths are compared with 5-fold cross-validation. Their average validation scores are 0.81 (weak), 0.86 (medium) and 0.84 (strong), and their training-fold scores are 0.99, 0.90 and 0.85. Which penalty should you pick?

    • Weak, because 0.99 is the best training-fold score
    • Strong, because the simplest model is always safest
    • Medium, because 0.86 is the best average validation score
    • None of them, because the training and validation scores disagree

  8. After picking a penalty by 5-fold cross-validation, Priya scores the untouched held-back set once, dislikes the number, changes the penalty, and scores the held-back set again. The second score is still an honest estimate of future performance.

    Circle one:   True   False

  9. You have hourly electricity readings for two years and want an honest estimate of next week's demand. You shuffle all the hours into 5 folds. What goes wrong?

    • Nothing goes wrong, because shuffling is exactly what keeps every fold representative
    • Five folds are too few for two years of hourly readings
    • Hours from after each check period end up in training, so the model peeks ahead
    • The readings must be split by day of the week instead

  10. What is the correct split for a time-ordered dataset?

    • Shuffle all rows and split at random
    • Train on the newest rows, then validate on the oldest ones
    • Train on the older rows, validate on the rows after them
LightMySky · lightmysky.comW1-mt_YG7X3HYCrT-s1

Answer key

For grown-ups. Fold this page away before handing over the rest.

Folds for choosing, one vault for judging W1-mt_YG7X3HYCrT-s1

  1. 4 trainings, each on 3 folds with the fourth fold as the check · Each of the 4 folds takes one turn as the check while the other 3 train, so there are 4 trainings and each uses 3 folds.
  2. 5 times · Each of the 5 folds takes one turn as the check, so there are 5 fresh trainings.
  3. True · One scoring at the very end keeps the reported number believable.
  4. 200 · The rows split evenly, so each fold holds 1000 divided by 5.
  5. False · The file is sorted by region, so each quarter is one stretch of the sorted order. With several regions the four folds cannot all match the whole dataset's mix of regions: the first quarter, for one, holds only the region or regions that sort first. With no time order, the lesson's rule is to assign each row to a fold at random, which drops file position from the choice of fold.
  6. Train on 2019 to 2023 and validate on 2024 · Cutting at a point in time keeps training in the past and validation in the future, which mirrors predicting 2025 from what came before.
  7. Medium, because 0.86 is the best average validation score · The choice rests on the validation average, and 0.86 is the best of the three. Training-fold scores favour the least penalised model, which is the one that memorises most.
  8. False · The second run used the held-back set to make a choice, so its score now reflects that choice. The set stays honest only when it is scored once, after every decision is fixed.
  9. Hours from after each check period end up in training, so the model peeks ahead · The readings are time-ordered. Shuffling puts later hours into the training folds, so the model learns from a future it would never see when predicting next week, and the score flatters.
  10. Train on the older rows, validate on the rows after them · Cutting at a point in time mirrors real use: past trains, future judges.
Worksheet · LightMySky