Reproducibility: Seeds, Environments and the Ablation · seed 1 · A4, ink-friendly. The answer key prints on its own page for grown-ups.

Making results repeatable and proving the gain

Computing · Software Engineering · ages 23-24
Name ______________________   Date ____________

  1. What does an ablation change?

    • Exactly one component
    • Everything at once
    • Nothing at all

  2. A stranger wants to repeat your run. What do they need?

    • Only the final score
    • Only the paper title
    • Seeds, environment, and the exact command

  3. A single seed counts as a full result.

    Circle one:   True   False

  4. Two runs differ in three settings at once. Can you claim one part caused the gain?

    • Yes, the best score always explains itself
    • Yes, three changes count as one
    • No, only one thing may differ

  5. Your score rose after adding a part. How do you show the part caused it?

    • Rerun with a new dataset and new seed
    • Rerun with only that part removed and compare
    • Report only the higher score

  6. You ran five seeds with spread-out scores. What do you report?

    • The scores across seeds, including how they vary
    • Only the best seed
    • The seed you liked most

  7. A teammate records the command but not the seed or environment. Repeatable?

    • Yes, the command is the only thing needed
    • Yes, seeds never matter
    • No, the seed and environment are required too

  8. A reviewer asks for the ablation isolating your new loss term. What do you run?

    • A new model with a new dataset
    • Same setup and seed, term on versus off, then compare the gap
    • One run with the term on only
LightMySky · lightmysky.comW1-mt_Rp__eJ-vjn-s1

Answer key

For grown-ups. Fold this page away before handing over the rest.

Making results repeatable and proving the gain W1-mt_Rp__eJ-vjn-s1

  1. Exactly one component · One difference per ablation, nothing more.
  2. Seeds, environment, and the exact command · Repeatable means every ingredient recorded.
  3. False · One seed may be luck; a result needs several.
  4. No, only one thing may differ · Tangled changes prove nothing about one part.
  5. Rerun with only that part removed and compare · The on-versus-off gap is the part's contribution.
  6. The scores across seeds, including how they vary · Report the finding and its wobble.
  7. No, the seed and environment are required too · All three ingredients are mandatory.
  8. Same setup and seed, term on versus off, then compare the gap · Same everything, one switch flipped, gap measured.
Worksheet · LightMySky