Cleaning Real Data: Missing, Duplicated and Wrong
Real datasets have blanks, repeats, impossible values and three spellings of the same city. Every repair is a decision that changes the result, so a cleaning step has to be written down and defended rather than done quietly.
What a learner can do afterwards
- Decide between dropping, filling and flagging a column of missing values, with a reason
- Find duplicates that are not exact copies and choose a matching rule
- Say how a cleaning choice could change the conclusion of the analysis
1 · Read
Real data has blanks, repeats, impossible values, and three spellings of one city. Every repair changes the result, so you write each cleaning step down and defend it rather than tidying quietly.
A city column reads Springfield, springfield, and SPRINGFIELD. You set one matching rule, lower case and trim spaces, and the three collapse into one city. A stable id number never shifts like spelling does, so rows with different ids stay two things.
For missing values you choose drop, fill, or flag, and say why. A blank tag is as broken as a repeated one, so rows missing their key need a plan. Dropping fits when the gaps are few and unrelated to the question, while dropping a third of the rows can move the answer.
Before you clean, ask how each choice could move the conclusion. Then write the rule down, because a different rule can give a different answer.
Match near duplicates with one written rule, handle blanks by drop fill or flag, and show how each choice moves the answer.
2 · Watch
Take it off screen
Where it sits
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.