Data Pipelines: Batch, Streaming and Doing It Twice · seed 1 · A4, ink-friendly. The answer key prints on its own page for grown-ups.

Moving data without breaking it

Computing · Data & Databases · ages 21-22
Name ______________________   Date ____________

  1. What does a batch job do with a record that arrives late?

    • It drops the record silently
    • It stops the whole pipeline
    • It recomputes the window with the record included

  2. What does it mean for a stage to be safe to run twice?

    • It runs in half the time
    • It gives the same result on the same input
    • It never needs monitoring

  3. A renamed field upstream can break every stage downstream of it.

    Circle one:   True   False

  4. A stream processor sees an order dated three days back. What is a deliberate handling?

    • Crash so someone notices
    • Pretend the date is today
    • Update the earlier result or route it to a side output

  5. A load appends rows with no keys and reruns after dying halfway. What happens?

    • Finished rows are written a second time
    • The rerun skips finished rows by itself
    • The source data is deleted

  6. Upstream renames total_price to amount. What should the team have done first?

    • Versioned the schema and agreed the change with downstream owners
    • Renamed it back in secret
    • Deleted all downstream stages

  7. A nightly load must survive dying halfway. Which design fits?

    • Keyless appends with a faster machine
    • Manual reruns watched by an engineer
    • Writes keyed by order id so reruns overwrite identically

  8. A teammate says exactly-once delivery alone makes retries safe. What is wrong?

    • Retries never happen in practice
    • Delivery promises do not fix a stage that doubles rows on rerun
    • Delivery order is what matters, not duplicates
LightMySky · lightmysky.comW1-mt_o0zBEPLbmF-s1

Answer key

For grown-ups. Fold this page away before handing over the rest.

Moving data without breaking it W1-mt_o0zBEPLbmF-s1

  1. It recomputes the window with the record included · Batch recomputes the full window, so the late record joins in.
  2. It gives the same result on the same input · A safe rerun repeats the same outcome instead of doubling it.
  3. True · Downstream stages read that field, so the rename breaks them.
  4. Update the earlier result or route it to a side output · Streaming answers late data by revising results or isolating the record.
  5. Finished rows are written a second time · With no keys there is nothing to overwrite, so rows duplicate.
  6. Versioned the schema and agreed the change with downstream owners · A versioned contract lets every reader adapt before anything breaks.
  7. Writes keyed by order id so reruns overwrite identically · Keys turn the rerun into harmless overwrites of the same rows.
  8. Delivery promises do not fix a stage that doubles rows on rerun · Safety comes from the stage giving the same result twice, not from delivery alone.
Worksheet · LightMySky