The shape of the failure
We pulled every failed run from a single quarter and sorted them by the step index where they stopped. The distribution was not flat. Failures clustered in the back half of longer graphs, and the single most common cause was a tool call that neither succeeded nor returned an error — it simply took longer than anyone had thought to bound.
A model producing a wrong answer is a visible, debuggable event. A rate-limited HTTP call that hangs for four minutes is not. The run is technically alive the whole time, which is exactly why nobody gets paged.
Timeouts are not a retry policy
The first instinct is to add a timeout, and it helps. But a timeout without a retry policy converts a hang into a failure, and a failure without a checkpoint converts into starting over. Teams then discover that starting over is not free: the first six steps already sent an email, wrote a row, and charged a card.
The three pieces are inseparable. A bound on how long a step may take, a rule for what to do when it exceeds that, and a record of what has already happened so the retry knows where to begin.
What we changed
We moved checkpointing from the end of a run to after every tool call. The write cost is real — roughly four milliseconds at the median — and it is the best four milliseconds in the system, because it turns every failure from a restart into a resume.
The number we watch now is not success rate. It is the ratio of resumed runs to restarted ones. When that ratio is high, failures have become cheap.