Solutions
Data pipelines
Long extractions that must survive a deploy, a rate limit, and a restart.
6hlongest run completed without restart
The challenge
A six-hour extraction that restarts from zero at hour five is not a slow pipeline, it is a broken one. Most schedulers treat the whole job as the unit of retry, which is the wrong granularity by several orders of magnitude.
What they built
Checkpoint after every batch, not every job. A restart resumes at the last committed offset, and an idempotency key derived from that offset means a duplicated batch writes the same rows rather than doubling them.
- 01Read the source in bounded batches with a committed offset
- 02Checkpoint before marking each batch complete
- 03Retry rate-limited calls per policy without failing the run
- 04Publish completion to a topic once, guaranteed