Training Data: Where It Came From and What It Leaves Out
What a model can do is bounded by what it was shown. Who is in the data, who is missing, when it was collected and whose permission was asked all decide what the model is fit for.
What a learner can do afterwards
- Name three things about a dataset that change what a model trained on it can be trusted with
- Give a case where a group missing from the data shows up later as worse performance for those people
- Say what a cut-off date means for questions about recent events
1 · Read
A model is bounded by what it was shown. Before trusting it, ask three things: who is in the data, who is missing, and when it was collected. Also ask whose permission was sought for those records.
Researchers once built a fitness model from males in one athletic group, then used it on everyone. It guessed worse for women and girls, because their patterns were never in the data. A whole group missing at training shows up later as worse results for those people.
Mind the cut-off date too. A model knows nothing past the day collection ended, so it cannot speak on later events. Questions about recent events fall outside what it was fitted on.
Watch for the wrong lesson, like the tank story: blurry photos of some tanks and sharp photos of others taught the model photo quality, not tanks. A separate test set catches such tricks before real use.
Check who is in, who is out, and how old the data is.
2 · Watch
Take it off screen
Where it sits
Learn first
This opens up
Where this leads
Jobs that lean on this skill. Follow one to see everything it is built on.
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.