Folds for choosing, one vault for judging

Computing · Machine Learning · ages 20-21

Name ______________________ Date ____________

You run 5-fold cross-validation on a dataset. How many times is a model trained from scratch?
- 1 time
- 5 times
- 10 times
- 50 times
You run 5-fold cross-validation. How many times is a model trained from scratch?
- 1 time
- 10 times
- 5 times
After cross-validation picks your model and settings, you should score the final untouched set once and report that number.
Circle one: True False
A dataset has 1000 rows. In 5-fold cross-validation, how many rows are in each validation fold?
Answer: ______________
When the rows in a dataset have no time order, each row should be assigned to a fold at random rather than by position in the file.
Circle one: True False
What is the correct way to split a time-ordered dataset for honest validation?
- Shuffle all rows and split at random
- Train on the older rows and validate on the rows that come after them
- Train on the newest rows and validate on the oldest
- Put every second row in validation
You are choosing a penalty strength for a regression model using 10-fold cross-validation. What should you pick?
- The penalty with the best average score across the 10 validation folds
- The penalty that scores best on the training folds
- The smallest penalty so the model fits training data exactly
- The largest penalty so the model stays simplest
After cross-validation picks your model and settings, you should score the final untouched test set once and report that number.
Circle one: True False
You have daily temperature readings for five years and want an honest estimate of next-month prediction. Why is a random split a bad idea?
- Random splits are always slower to compute
- Future days would appear in training, so the model could cheat by peeking ahead
- Random splits always give folds of unequal size
- Random splits do not work with numeric data
What is the correct split for a time-ordered dataset?
- Shuffle all rows and split at random
- Train on the newest rows, validate on the oldest
- Train on the older rows, validate on the rows after them

Answer key

For grown-ups. Fold this page away before handing over the rest.

Folds for choosing, one vault for judging W1-mt_YG7X3HYCrT-s1

5 times · In 5-fold cross-validation, each of the 5 folds takes a turn as the validation fold, so the model is trained 5 separate times, each on the other 4 folds.
5 times · Each of the 5 folds takes one turn as the check, so there are 5 fresh trainings.
True · One scoring at the very end keeps the reported number believable.
200 · The rows split evenly, so each fold holds 1000 divided by 5.
True · Random assignment keeps each fold representative, so every fold gives a fair check on the model. Assigning by position can concentrate similar rows in one fold and skew the scores.
Train on the older rows and validate on the rows that come after them · Cutting at a point in time keeps training in the past and validation in the future, which mirrors how the model will actually be used.
The penalty with the best average score across the 10 validation folds · Cross-validation averages validation scores across folds, and the penalty with the best average is the one most likely to perform well on new data. Training-fold scores favor overfitting.
True · The held-back test set stays believable only if it is scored a single time, after all choices are locked in. Repeated scoring turns it into a selection tool.
Future days would appear in training, so the model could cheat by peeking ahead · In time-ordered data, a random split puts future rows into training. The model then learns from information that would not exist at prediction time, and the score flatters you.
Train on the older rows, validate on the rows after them · Cutting at a point in time mirrors real use: past trains, future judges.