Executive Summary
- Long-running tasks cannot rely on in-memory process state; server restarts and deployments will terminate them.
- A formal state machine (QUEUED, RUNNING, CHECKPOINTED, WAITING, RESUMING, COMPLETED) governs the task lifecycle.
- Distributed checkpointing writes flattened state snapshots to disk after every tool execution step.
- Deterministic replay reconstructs agent working state while skipping already-committed external side effects.
- Idempotency keys on all mutating tools prevent duplicate API calls and double-billing during automatic retries.
The reality of multi-hour agent workflows
In production software engineering, tasks like migrating a legacy database, conducting a comprehensive security vulnerability scan, or reconciling a quarter's worth of merchant invoices cannot be completed in a single prompt loop. These tasks involve dozens of sequential tool invocations and can run for hours.
Over a three-hour execution window, things will go wrong: Kubernetes worker nodes will be evicted, network switches will drop packets, third-party APIs will hit rate limits, and language model providers will experience transient 500 errors.
If an agent holds its state only in local memory, any interruption wipes out progress, forcing a complete restart that wastes tokens and risks executing duplicate mutations. Distributed reliability requires building tasks as durable state machines.
Assume process death at any moment
Design agent workers with the assumption that the host process will be killed after every tool execution. If the task cannot resume cleanly from disk in 100 milliseconds, the architecture is not production-ready.
The durable task state machine lifecycle
A reliable long-running agent must be governed by an explicit finite state machine. Rather than an unbounded loop, execution transitions through distinct, verifiable states.
1. QUEUED: Task intention is registered in the persistent queue with validated input parameters and an allocated token budget.
2. RUNNING: A worker claims the task, acquires a distributed lock, and executes the active step in the plan DAG.
3. CHECKPOINTED: The worker commits the step's observation, updates the event log, and persists the memory snapshot to disk.
4. WAITING: The task suspends itself while waiting for an external timer, webhook callback, or Human-in-the-Loop approval.
5. RESUMING: A worker picks up the suspended task, verifies the lock, replays past events, and resumes execution.
6. COMPLETED / FAILED: Task terminates with a final verified output payload or escalates to a human with a diagnostic report.
Durable task state machine transitions
State transitions are committed to a Write-Ahead Log (WAL) before external tool side effects are triggered.
Checkpointing and WAL persistence
Checkpointing is the foundation of durable execution. After every completed action, the execution engine writes a state snapshot to a persistent Write-Ahead Log (WAL) backed by PostgreSQL or Redis.
The snapshot contains the task ID, sequence number, working token context, the completed plan DAG nodes, and the current environment variables.
By enforcing strict serialization, the system decouples the stateless LLM worker from the durable task state. If a worker pod crashes mid-task, another worker pod immediately claims the task from the queue and rehydrates state from the last committed checkpoint.
State persistence strategies compared
| Feature | Strategy | In-Memory Process | Periodic Checkpointing | Event-Sourced WAL |
|---|---|---|---|---|
| Interruption Recovery | Impossible (State lost) | Resumes from last checkpoint (Minutes lost) | Exact recovery with zero lost state | |
| Replay Accuracy | Zero determinism | Partial replay from snapshot | Deterministic replay of all events | |
| Database Overhead | Zero DB writes | Low (snapshot written every N steps) | Moderate (every event appended to log) | |
| Production Suitability | Demos & single-prompt bots | Medium-complexity tasks (< 10 min) | Mission-critical enterprise workflows (Hours/Days) |
Durable worker TypeScript code contract
The TypeScript contract below defines a durable task worker interface with checkpointing, heartbeat management, and recovery hooks.
Idempotency keys and side-effect deduplication
When an interrupted task is resumed, it may need to retry an action that was in-flight during the crash. If that action involved transferring funds, sending an email, or spinning up cloud servers, retrying without safeguards will cause duplicate side effects.
We solve this with Idempotency Keys. Every mutating tool call generated by the agent must include a deterministic idempotency key derived from the `taskId` and `sequenceId` (e.g. `idem_task882_step4`).
The external tool or API gateway checks this key in a Redis deduplication table before executing. If the key has already been processed, the gateway returns the cached output from the previous call instead of re-executing the mutation.
Handling timeouts, pauses, and worker crashes
Distributed task systems must handle worker crashes cleanly. We maintain a heartbeat mechanism: every running worker must ping the task coordinator every 15 seconds.
If a worker stops emitting heartbeats for more than 45 seconds (indicating a process crash or node failure), the task coordinator marks the worker as dead, breaks the distributed lock, and re-queues the task with a `RESUMING` status.
The next available worker claims the task, loads the latest checkpoint from PostgreSQL, checks idempotency keys, and resumes execution seamlessly.
Worker failure and automatic recovery sequence
Long-running task reliability checklist
Verify these distributed reliability controls before launching multi-hour agent workflows.
Reliability engineering checklist
1State Machine & Checkpointing
- Tasks are governed by explicit finite state machines
- State snapshots are written to PostgreSQL/Redis after every tool step
- Worker crashes can resume from disk in under 100 milliseconds
2Idempotency & Deduplication
- Deterministic idempotency keys are attached to all mutating tool calls
- API gateways deduplicate requests using a distributed Redis cache
- Replay engine skips already-committed external side effects
3Fault Tolerance & Heartbeats
- Heartbeat monitors detect dead workers and re-queue orphan tasks
- Transient tool failures trigger exponential backoff retry with jitter
- Budget circuit breakers enforce strict max runtime and token caps