Regularisation: Charging for Complexity · seed 1 · A4, ink-friendly. The answer key prints on its own page for grown-ups.

A charge for big numbers

Computing · Machine Learning · ages 20-21
Name ______________________   Date ____________

  1. Which penalty can land some coefficients exactly on zero?

    • The squared penalty
    • No penalty can do that
    • The absolute penalty

  2. What happens to the coefficients as the penalty grows?

    • They grow without limit
    • They shrink toward zero
    • They freeze immediately

  3. Features must be scaled before a penalty is fair.

    Circle one:   True   False

  4. Why must features be scaled before penalising?

    • Scaling speeds up the computer clock
    • Small numbers force big coefficients
    • Scaling removes the need for any penalty on its own

  5. Your regression fits training data perfectly with giant coefficients but flops on new data. What do you add, and what should you see?

    • A penalty, with smaller coefficients and better new-data error
    • More features, and even bigger coefficients
    • A smaller training set, leaving the coefficients exactly as they were

  6. You need automatic feature selection, not just smaller weights. Which penalty and why?

    • Absolute, because exact zeros drop features out
    • Squared, because smooth shrinking selects features
    • Neither, because penalties cannot select

  7. Someone penalises without scaling: one feature in kilometres dominates the penalty and gets crushed, though it predicts well. What went wrong?

    • The penalty was too small to matter
    • Units decided the charge, so a good feature was punished
    • Absolute penalties always crush the features that predict well

  8. After an absolute penalty, three of ten coefficients are exactly zero. A teammate says those three features were useless all along. What is the careful reading?

    • They were useless, and the proof is final
    • The penalty failed here, since zeros can only ever mean errors
    • They added nothing given the rest, at this penalty strength
LightMySky · lightmysky.comW1-mt_RVgsDP0XzP-s1

Answer key

For grown-ups. Fold this page away before handing over the rest.

A charge for big numbers W1-mt_RVgsDP0XzP-s1

  1. The absolute penalty · Its sharp corner at zero lets coefficients land on it exactly.
  2. They shrink toward zero · A bigger charge on size pulls the coefficients back as a whole.
  3. True · Without scaling, the units decide the penalty instead of usefulness.
  4. Small numbers force big coefficients · The same length in kilometres needs a coefficient a thousand times bigger than in metres, so an unscaled penalty charges for units.
  5. A penalty, with smaller coefficients and better new-data error · The penalty trades a little training fit for better new-data error.
  6. Absolute, because exact zeros drop features out · Exact zeros remove features from the model on the way.
  7. Units decided the charge, so a good feature was punished · Kilometres give small numbers, so the feature needed a big coefficient and paid a big penalty. The charge followed the units.
  8. They added nothing given the rest, at this penalty strength · Zero means unneeded beside the others, not useless in every possible model.
Worksheet · LightMySky