Data Pipelines: Batch, Streaming and Doing It Twice
Data moves from where it is produced to where it is analysed through stages that extract, transform and load it. Any stage can be retried after a failure, so each one has to be safe to run twice, and late or out-of-order records have to be handled on purpose.
What a learner can do afterwards
- Design a stage that gives the same result when it is run twice on the same input
- Say what a batch job and a stream processor each do with a record that arrives late
- Explain what a schema change upstream does to everything downstream
1 · Read
Data travels from where it is produced to where it is analysed through stages that extract, transform and load it. Any stage can fail halfway and be retried, so each stage must give the same result when it runs twice on the same input.
A nightly load writes each order under its order id. When the job dies halfway and reruns, the finished rows are simply overwritten with the same values instead of being counted twice. Keys make the rerun safe.
Records also arrive late or out of order, and each style of job answers on purpose. A batch job recomputes the whole window with the late record included. A stream processor updates its earlier result or routes the record to a side output.
Treat the upstream schema as a contract. A renamed field upstream breaks every stage below it, so version the schema and agree on changes before they ship.
Make every stage safe to rerun, handle late records on purpose, and guard the schema your pipeline rests on.
2 · Watch
Take it off screen
Where it sits
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.