Data Pipelines: Batch, Streaming and Doing It Twice · seed 1 · A4, ink-friendly. The answer key prints on its own page for grown-ups.

Moving data without breaking it

Computing · Data & Databases · ages 21-22
Name ______________________   Date ____________

  1. A nightly batch job reports sales totals by order date. When a record arrives late, the job recomputes the window for that record's order date. Monday's total stood at 40 sales. On Wednesday, one more sale dated Monday arrives. After that night's run, what is Monday's total?

    • 40, and the sale counts toward Wednesday instead
    • 40, because Monday's window has already closed
    • 41, with the late sale included

  2. What does it mean for a stage to be safe to run twice?

    • It never fails, so it never needs a second run
    • A second run leaves the same result as one run
    • It never needs monitoring

  3. A renamed field upstream can break every stage downstream of it.

    Circle one:   True   False

  4. A stream processor sees an order dated three days back. What is a deliberate handling?

    • Crash so someone notices
    • Stamp it with today's date so it lands in the current window
    • Update the earlier result or route it to a side output

  5. A load appends rows with no keys and reruns after dying halfway. What happens?

    • Finished rows are written a second time
    • The rerun skips finished rows by itself
    • The source data is deleted

  6. Upstream renames total_price to amount. What should the team have done first?

    • Versioned the schema and agreed the change with downstream owners
    • Renamed it back in secret
    • Deleted all downstream stages

  7. An hourly job copies payments into a warehouse and must survive dying halfway. Which design fits?

    • Keyless appends, rerun from the start after each failure
    • Manual reruns watched by an engineer
    • Writes keyed by payment id so reruns overwrite identically

  8. A teammate says exactly-once delivery alone makes retries safe. What is wrong?

    • Retries never happen in practice
    • Delivery promises do not fix a stage that doubles rows on rerun
    • Delivery order is what matters, not duplicates
LightMySky · lightmysky.comW1-mt_o0zBEPLbmF-s1

Answer key

For grown-ups. Fold this page away before handing over the rest.

Moving data without breaking it W1-mt_o0zBEPLbmF-s1

  1. 41, with the late sale included · The job recomputes the window for the sale's order date, Monday, so Monday's total becomes 41. The sale does not go into Wednesday's total, because totals follow order date, not arrival date.
  2. A second run leaves the same result as one run · A safe rerun repeats the same outcome instead of doubling it.
  3. True · Downstream stages read that field, so the rename breaks them.
  4. Update the earlier result or route it to a side output · Streaming answers late data by revising results or isolating the record.
  5. Finished rows are written a second time · With no keys there is nothing to overwrite, so rows duplicate.
  6. Versioned the schema and agreed the change with downstream owners · A versioned contract lets every reader adapt before anything breaks.
  7. Writes keyed by payment id so reruns overwrite identically · Keys turn the rerun into harmless overwrites of the same rows.
  8. Delivery promises do not fix a stage that doubles rows on rerun · Safety comes from the stage giving the same result twice, not from delivery alone.
Worksheet · LightMySky