Skip to content
Apkar

Engineering

A short primer on durable execution

Checkpoints, idempotency keys, and why exactly-once is a promise about effects, not messages.

Omar Haddad · Infrastructure9 min read

Durability is about effects

Exactly-once delivery is not achievable in a distributed system, and chasing it is how teams lose a quarter. Exactly-once effects are achievable, and that is what actually matters: the email goes out once even if the message arrives three times.

The mechanism is an idempotency key derived from the step identity rather than generated per attempt. Retries then collapse into the same effect instead of stacking.

Checkpoint before you commit

Ordering matters more than frequency. A checkpoint written after a step is marked complete will, on a cold start, replay a step that already ran. Writing it before means a crash costs you a duplicate attempt at worst — and the idempotency key absorbs that.

This is the bug that took us longest to find, because it only appeared during deploys, which is exactly when nobody is reading logs carefully.