Building Long-Running Agents

@dalexeenko|July 1, 2026 (4w ago)59 views

Several weeks ago I built Ants and wrote about it: the loop, SSE streaming, sandboxing, verifiers. It was mostly exploring ideas about how to make agents work. Since then I've been thinking about how to make agents survive and provide more reliable execution.

The Ants harness ran the loop in-process: you POST a prompt, a background task spins up a ReAct loop, and events stream back over SSE. It works fine until the process restarts: when that happens, the agent that was a few tool calls into a ten-minute task loses all of its work. For a background agent that's a deal breaker.

So how do we fix it?

Separate the control plane from execution

The first thing is to have a hard split between two layers: a control plane — an API that accepts prompts, starts and aborts runs, and streams events — and worker execution, the thing that actually grinds through LLM calls and tool batches.

The glue between them is a single interface, RunExecutor, with two implementations: InProcessRunExecutor and TemporalRunExecutor (this is the durable one).

Browser / CLI
     │
     ▼
Control plane (API) ──────▶ Postgres (runs · events)
     │                            ▲
     ▼                            │ append / replay
RunExecutor ──┬── InProcess       │
              └── Temporal ───────┘
                    │
                    ▼
               Tool batch ──▶ discovery ──▶ remote daemon(s)

Durable execution is about shape

The lesson I underestimated: making a loop durable is less about the durability engine and more about how you shape the loop. To run the loop as a durable workflow, I had to break it into discrete steps:

appendUserMessage → runAssistantTurn → runToolBatch → repeat → finalizeRun

Once you decompose the loop, everything becomes simpler: say a worker crashes, then another picks up the workflow and replays to exactly where it was.

This approach to looping is important, even before building a durability engine. Forcing yourself to name every step and make it resumable is the kind of constraint that improves the entire design.

Streaming

In the previous harness, Ants, streaming and execution were mixed in: the loop pushed events straight to the SSE response. That can't work when the executor is a worker on another machine and the client is talking to a stateless API.

So I decoupled events from execution entirely. Activities append to a run event store in Postgres. The API's job is just to replay that store to whoever connects and tail new events as they land. If you close the tab and reopen it an hour later, you'll still get the full history plus the live stream — because the stream was never tied to the connection in the first place.

Remote tools

The other thing is to run tools somewhere other than the agent server, i.e., a remote execution environment. You can have those execution environments connect back and receive tool calls over SSE. You need to build sticky sessions, routing, and failover, and keep them out of the durable-execution layer, because they're different concerns that fail in different ways.

If a daemon that runs the remote execution dies mid-batch, the natural step is to retry the tool call that was in progress. But that call might have already written a file, sent an email, or hit an API — retrying may have side effects. So the harness restarts the whole tool batch on a replacement daemon instead of resuming a partial one.

Retry budgets

Four retry mechanisms ended up stacked on each other, each added for a good local reason:

| Layer | Budget | | --- | --- | | Temporal activity retry | 3 attempts | | Tool batch recovery | 1 restart | | Loop detection window | 5 identical batches | | Max iterations | 12 |

A single prompt can, in the worst case, make several dozen LLM calls before anything fails — it takes time and it's expensive.

If you stack retries, reason about the budget as a single number.

We haven't won yet

A few things aren't quite ready yet, I'll be writing about those soon.