---
title: "Cleaning Real Data: Missing, Duplicated and Wrong"
description: "Real datasets have blanks, repeats, impossible values and three spellings of the same city. Every repair is a decision that changes the result, so a cleaning step has to be written down and defended r"
canonical: https://lightmysky.com/learn/computing/cleaning-real-data-missing-duplicated-and-wrong-mt_qIsLXA-7EC
source: https://lightmysky.com/learn/computing/cleaning-real-data-missing-duplicated-and-wrong-mt_qIsLXA-7EC.md
retrieved: 2026-09-12
---

> **Agent view.** This is the Markdown twin of the page, for tools and assistants.
> When to use this site, and the call that answers each job: https://lightmysky.com/agent-instructions.md
> API description (OpenAPI 3.1): https://lightmysky.com/openapi.json · Authentication: https://lightmysky.com/auth.md
> Pricing: https://lightmysky.com/pricing.md · Catalog: https://lightmysky.com/llms.txt · Full catalog: https://lightmysky.com/llms-full.txt
> Every machine-readable file on this domain: https://lightmysky.com/.well-known/ai-catalog.json
> Ask for Markdown with `Accept: text/markdown`, a `.md` address, or `?mode=agent`.

# Cleaning Real Data: Missing, Duplicated and Wrong

Real datasets have blanks, repeats, impossible values and three spellings of the same city. Every repair is a decision that changes the result, so a cleaning step has to be written down and defended rather than done quietly.

Subject: Computing · Area: Data & Databases · Ages 21 to 22
Page: https://lightmysky.com/learn/computing/cleaning-real-data-missing-duplicated-and-wrong-mt_qIsLXA-7EC

## Ready when they can

- Decide between dropping, filling and flagging a column of missing values, with a reason
- Find duplicates that are not exact copies and choose a matching rule
- Say how a cleaning choice could change the conclusion of the analysis

## Lesson: Clean it and write down the rule

Real data has blanks, repeats, impossible values, and three spellings of one city. Every repair changes the result, so you write each cleaning step down and defend it rather than tidying quietly.

**Example.** A city column reads Springfield, springfield, and SPRINGFIELD. You set one matching rule, lower case and trim spaces, and the three collapse into one city. A stable id number never shifts like spelling does, so rows with different ids stay two things.

For missing values you choose drop, fill, or flag, and say why. A blank tag is as broken as a repeated one, so rows missing their key need a plan. Dropping fits when the gaps are few and unrelated to the question, while dropping a third of the rows can move the answer.

**Tip.** Before you clean, ask how each choice could move the conclusion. Then write the rule down, because a different rule can give a different answer.

**Recap.** Match near duplicates with one written rule, handle blanks by drop fill or flag, and show how each choice moves the answer.

## Practice

8 questions on this page, each with its working shown.

## Needs first

- [The Data Science Workflow](https://lightmysky.com/learn/computing/the-data-science-workflow-mt_99QNfiyQ55)
- [Reading and Writing Files, Including CSV](https://lightmysky.com/learn/computing/reading-and-writing-files-including-csv-mt_j5qb4FdkdI)

## Opens up

- [Exploratory Analysis and an Honest Chart](https://lightmysky.com/learn/computing/exploratory-analysis-and-an-honest-chart-mt_Jpb78Z6Ytu)