---
title: "Data Pipelines: Batch, Streaming and Doing It Twice"
description: "Data moves from where it is produced to where it is analysed through stages that extract, transform and load it. Any stage can be retried after a failure, so each one has to be safe to run twice, and "
canonical: https://lightmysky.com/learn/computing/data-pipelines-batch-streaming-and-doing-it-twice-mt_o0zBEPLbmF
source: https://lightmysky.com/learn/computing/data-pipelines-batch-streaming-and-doing-it-twice-mt_o0zBEPLbmF.md
retrieved: 2026-09-12
---

> **Agent view.** This is the Markdown twin of the page, for tools and assistants.
> When to use this site, and the call that answers each job: https://lightmysky.com/agent-instructions.md
> API description (OpenAPI 3.1): https://lightmysky.com/openapi.json · Authentication: https://lightmysky.com/auth.md
> Pricing: https://lightmysky.com/pricing.md · Catalog: https://lightmysky.com/llms.txt · Full catalog: https://lightmysky.com/llms-full.txt
> Every machine-readable file on this domain: https://lightmysky.com/.well-known/ai-catalog.json
> Ask for Markdown with `Accept: text/markdown`, a `.md` address, or `?mode=agent`.

# Data Pipelines: Batch, Streaming and Doing It Twice

Data moves from where it is produced to where it is analysed through stages that extract, transform and load it. Any stage can be retried after a failure, so each one has to be safe to run twice, and late or out-of-order records have to be handled on purpose.

Subject: Computing · Area: Data & Databases · Ages 21 to 22
Page: https://lightmysky.com/learn/computing/data-pipelines-batch-streaming-and-doing-it-twice-mt_o0zBEPLbmF

## Ready when they can

- Design a stage that gives the same result when it is run twice on the same input
- Say what a batch job and a stream processor each do with a record that arrives late
- Explain what a schema change upstream does to everything downstream

## Lesson: Moving data without breaking it

Data travels from where it is produced to where it is analysed through stages that extract, transform and load it. Any stage can fail halfway and be retried, so each stage must give the same result when it runs twice on the same input.

**Example.** A nightly load writes each order under its order id. When the job dies halfway and reruns, the finished rows are simply overwritten with the same values instead of being counted twice. Keys make the rerun safe.

Records also arrive late or out of order, and each style of job answers on purpose. A batch job recomputes the whole window with the late record included. A stream processor updates its earlier result or routes the record to a side output.

**Tip.** Treat the upstream schema as a contract. A renamed field upstream breaks every stage below it, so version the schema and agree on changes before they ship.

**Recap.** Make every stage safe to rerun, handle late records on purpose, and guard the schema your pipeline rests on.

## Practice

8 questions on this page, each with its working shown.

## Needs first

- [Message Passing and Asynchronous Work](https://lightmysky.com/learn/computing/message-passing-and-asynchronous-work-mt_twcpO-945r)
- [Partitioning and Replication for Scale](https://lightmysky.com/learn/computing/partitioning-and-replication-for-scale-mt_Xvzz5Hnuvu)

## Opens up

- [The Data Science Workflow](https://lightmysky.com/learn/computing/the-data-science-workflow-mt_99QNfiyQ55)
- [Stream Processing: Windows, Watermarks and Late Events](https://lightmysky.com/learn/computing/stream-processing-windows-watermarks-and-late-events-mt_CfzXL_cL0D)