Stream Processing: Windows, Watermarks and Late Events
An unbounded stream has no end at which to compute an answer, so results are cut by windows over event time. A watermark is the system's claim that event time has passed a point, and every record arriving after it forces a choice between waiting longer and being wrong.
What a learner can do afterwards
- Assign records to windows by event time and by processing time and show where the two differ
- Say what a watermark asserts and what it costs to hold one back
- Choose a policy for late records and defend it against the alternative
1 · Read
A stream has no end, so you cut answers by windows, which are fixed slices of time that each collect their own result. Event time is when each record happened in the world, while processing time is when your system happened to see it. The two agree while records arrive in order and on time, and they split apart whenever records arrive late or out of order.
A watermark is the system's claim that event time has passed a point, so windows up to that point can close and report. Holding the watermark back keeps windows open longer, which catches more late records at the price of slower answers. Emit early and you answer fast but risk results that late records prove wrong.
Picture a five-minute window counting taps on a video, with one phone sending its taps ten minutes late. If your policy drops late records, the published count stays fast and final but misses those taps. If your policy folds late records in with a correction, the count stays true but past answers must be revised.
Pick the late policy from what the answer is for. When past numbers must stand, like a public leaderboard, drop late records and say so. When truth matters more than stability, like billing, accept corrections and revise. Name the trade before you choose.
Window by event time, mark progress with watermarks, and pick a late policy that fits what the answer is for.
2 · Watch
Take it off screen
Where it sits
Learn first
This opens up
Nothing builds on it yet.
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.