After the Agent Crashes, Pick Up Where You Left Off

Part Two

20:36 What Is Pi Durable? Explaining Durable Agents in Five Sentences

Host: I don’t know — you can’t build applications with that thing. It’s just a hill climbing-type skill. It’s a bit disappointing. Host: I did want to ask you about Pi Durable. You used the word “durable” — what’s the rationale behind it? What’s actually going on?

Armin: Let me try to explain it in five sentences, okay? An agent starts in a process, the agent calls a tool, the process dies. What you want is: restart the process, and the agent picks up exactly where it left off.

Armin: That’s durability. You don’t have to manually type “continue,” the tool results and the assistant message stream don’t get lost, and your application state is durably saved and reliably recovered — that’s durability. Why do you need it? Because for long-running agents — anything Claw-like — you need this. The other problem Pi Durable solves is: previously, Pi would load the entire conversation, or really the entire conversation tree in Pi, into memory. If your Slack bot sits in your Slack for years, its full conversation history accumulates a huge pile of messages, and that obviously doesn’t work. These aren’t necessarily things you send to the LLM, but you still need to be able to access the full history. That’s basically what Pi Durable does. It’s also very easy to deploy in a lot of places — Vercel, Cloudflare DO, E2B, your laptop, my phone — which makes it quite interesting to use.

Host: Maybe you could walk me through its architecture a bit.

Mario: I actually read that repo today, and I also had a clanker (Claude) try to explain to me what was going on. My rough understanding of resumability is: you have to commit an action — at least that’s how I understand it. You have an action, and that action has to be committed to a file for it to be officially canonical. If it drops midway, you have something like an event queue.

22:28 The Effect Sandwich: Intent, Effect, Result

Mario: So there’s a bunch of events in the queue — at least that’s how I understand it. Right?

Armin: Right — and that’s exactly what we don’t do. We don’t do event sourcing. That’s what everyone else does. Uh-huh. We don’t do that because it has a bunch of problems. Instead, let me tell you how we do it, okay? The fundamental problem with durability is: you’re executing something, and it can crash at three points in time — before you start executing it, during execution, or after it’s done.

Armin: Right? And the thing you’re executing is usually called an effect. So most durability approaches do the following: before I execute it, I write something to some storage saying “I’m about to execute this thing with these parameters, these inputs,” right? So if I crash after saving that, I know I’ve correctly evaluated all the inputs, I’ve validated them, I’m ready to execute. If my process dies now — not executed yet, but the intent is saved —

Armin: I can restart, look at the database, and find there’s something “about to happen with these inputs.” Since it hasn’t been executed, I just do it, do it now, right? That’s the first part of the sandwich. The second place you can crash is while the effect — the execution — is actually running. This depends on what’s running. If it’s an external system, like you send to GitHub Actions and say “build this commit in this action,” right? You send it out.

Armin: Then your process dies, you bring it back up, and at that point you don’t necessarily know whether the thing actually executed, because you haven’t saved anything yet — you haven’t received the result back. You were cut off mid-conversation with GitHub, right? So how do you continue in that case? GitHub is easy here, because before the run starts, you can say “I want to schedule a run,” and you’ll get back a key. Before actually executing that run, you can save it first, right? An ID. So if I then go to execute that thing,

Armin: I say “execute using this ID,” right? If it’s already running, GitHub will tell me it’s already running — fine, then I wait, and you give me the result. If it hasn’t run yet, GitHub will start that run using that ID, right? This is called idempotency —

Mario: Armin, your favorite word — idempotency.

Armin: I fucking can’t pronounce it. Idempotency.

Mario: Right. So cool.

Armin: So if our external systems have this property — I give it an ID and say “do this thing with this ID,” and that thing is already running —

Armin: it tells me “I’m already doing it, wait for it to finish,” then I can survive after my effect crashes too, right? The final step is: once the effect has finished, I get a result, I save that result, and at that point my effect sandwich — the intent, the execution of the effect, and the storage of the effect’s result — is complete. After that, I never have to think about it again. I can throw away everything that happened, unless I need some trace for logging or auditing or whatever.

Armin: But the important thing is: I now have a result. That’s all I want. I want to reach that point regardless of whether my process crashed at some step in the sandwich. Now, there are complications. For example, if my effect is deleting a file on disk, right? I usually can’t undo that unless I have something like Git, right? If I charge your credit card, I don’t want to charge it twice. So all the effects you run, in some sense, also need to work with this durability system.

Armin: Most external systems can work with it, but some can’t. So you have to decide how to handle that. Back to Pi — the architecture of Pi Durable. Basically, everything that happens in the harness is a task, and each task is one of these effect sandwiches. For example, if I want to generate an LLM response, I have a sandwich: prepare the request, gather the context window and parameters, thinking level, model, etc. Save those, indicating “we’re going to do this now.”

Armin: Then actually call the LLM — that’s the effect. Then you get the response, and you persist the response. Tool calls work the same way. I get a tool call from the LLM, and I have a task to execute that tool call from the LLM. That’s the effect.

Mario: Right. I assume these nest inside each other, right?

Armin: Right. One task can spawn another task and wait for it to finish. That’s the entire magic, nothing more. The other side is the application state I mentioned.

27:11 Versioned Documents for Agent State

Armin: And what we built on that side is basically — you know MongoDB and document databases?

Mario: Yeah, sure.

Armin: It basically stores JSON objects, right? You can modify them freely. So for application state, we built a small document store with a special feature: I have a document, at timestamp zero of the transcript, the document has that value, right? Like a to-do list, right? Then you work with the agent — user, assistant, user, assistant, tool call — and it changes the to-do list.

Armin: So now the document has been updated at this position in the transcript. Now, if I want to go back in time, which version of the document do I want at that moment? This one or this one? It depends. Sometimes you want the version at that moment, right? I go back to this point in time and I want the old to-do list. Other times, I just want the latest version of the document, no matter where I am now. So our document store allows you to associate both types of document with

Armin: any position in your transcript. With this, you can store your application state. For example, I have a canvas I can draw on — it’s just a document storing a sequence of strokes, and I can replay those strokes to draw that picture, right? When I chat with my agent, draw on it, and then say “show the agent this new drawing,” that document evolves and changes along with the agent. If I go back in time, I want the old version of that drawing,

Armin: the one the agent was given, for example. With just two primitives — the versioned document and the task — I can do a whole lot of things. That’s basically Pi Durable.

Mario: So what you’re saying is, you have a versioned document, and as you progress through the session, that document gets updated, and each update is a version. Do you store the diffs between versions? Or…

Armin: Right, exactly. There’s a thing called JSON Patch — I think it’s an RFC, I’m not entirely sure.

29:31 Immutable State, CRDTs, and Proxies

Armin: That’s what we looked at initially. What it basically does is define various operations on JSON objects — like set this field to this value, create an array, set this element of the array to this value, etc. We looked at it — or rather, I looked at it — and found it wasn’t sufficient for this use case. So we built something similar but optimized for agentic scenarios.

Mario: I think my overall take is that I’ve been somewhat surprised — I’ve been doing web development for — I don’t know, maybe twenty years? Problems like partial — basically React’s core is state one, state two, how do you express those on immutable data structures? And somehow, it actually never really got dramatically better, which surprises me. I mean, there are things like Immer, right, which does a decent job.

Armin: Right, there has been some progress, but it’s not like there’s a widely-recognized, mature standard for expressing structured updates to nested data structures.

Mario: And then there’s Automerge, the project from those people at Ink & Switch, which tries to give you some kind of proxy object that records all the changes you make to it, and then generates the CRDT needed to express those changes underneath. But the way it maps to JavaScript is just a big hack, full of bugs.

Armin: Right. Software development, right? I mean, in Pi Durable there are obviously some footguns related to this too —

Armin: We also use the proxy approach. What does that mean? Basically, I tell the store “give me this document,” you get a JavaScript object, it looks like a JavaScript object, you can manipulate it like any other JavaScript object. I can write object.x = 10, and this gets automatically recorded as an operation and applied to the actual object. You can replay it to do this, right? That’s also done with a proxy, which is frankly a hack, a very slow hack.

31:15 Why Build a Durable Agent Harness

Armin: Exactly. But for what we need to do in an agentic harness, it works very well.

Mario: So there’s a huge problem space here, or a huge set of different opportunity areas. I think multiplayer is one of them. I haven’t really seen a good multiplayer harness. I’ve been trying to get my wife to use Hermes more so we have fewer issues coordinating things.

Host: Right, okay. But there just isn’t a good solution. The whole ecosystem is fragmented — I mean, there’s a new model every day.

Host: New stuff, everything is constantly changing. Why did you decide to solve this problem rather than some other agent harness problem?

Armin: Well, for me it’s —

Host: Do you mean multiplayer or durability?

Armin: I think both.

Host: Okay.

Armin: Durability, right, right — because as you just said, there really isn’t much off-the-shelf stuff out there. I mean, obviously there’s Hermes, OpenClaw, and various other claw-type things — there’s even AMP, right, with their orbs, which can transfer sessions.

Armin: And then keep running. They all need to solve the durability problem in some way, especially for agent harnesses. But none of those are general-purpose solutions — I can’t take them and build the next Earendil project on top, or the next open-source product, or some swarm-type agent system inside Shopify, right? So it’s hard to take these existing things and make them your own. What I wanted to create is a very small primitive. I mean, it’s only about twelve thousand lines of code.

Armin: And make it general enough and powerful enough to basically do everything an agent needs out of the box. That includes compaction and all that other stuff. And I think an important point to be clear about here: you obviously don’t need it. You absolutely don’t need Pi Durable — you could totally build your own.

Mario: You’re quite the marketer.

Armin: But we’ve had several projects where we built agents and actually got them to run in a way that made you trust them — they were reliable in some sense, they kept recovering from failures, but most importantly, they kept going even when no one was watching, no one was babysitting. Rather than “okay, it crashed now, should we continue?” Truly making it intrinsically reliable, making it a foundation you feel like you can build on — that alone is a ton of work. And it’s not about how many lines of code it ends up providing.

Armin: It’s about whether the core design really holds up. And as agents become more and more orchestrated, more capable of running independently of the human observing them and interacting with them, you’ll want — in a sense, they’re already unreliable enough: there’s an LLM, it does whatever it wants, right? It’s like some kind of organism. So the orchestration around it should at least be somewhat comprehensible.

Armin: And create some degree of sanity.

Mario: So I’ve been trying to use my laptop less — or my computer less. One thing is: closing the laptop. Even if you have things like caffeinate running, it’s still — it’s still annoying. So I’ve been trying to be able to send sessions back and forth between my different machines. I had GLM help me build something like this, which I call parcels — it’s built on Pi. The way it figured out on its own was —

Mario: This wasn’t my decision — it copies the JSON files that are the sessions from one machine to another, and just restarts, since the environment variables are all in there. Yeah, should — should be good enough. But it’s still just — I wish it could run on its own, spanning all my devices at the same time.

Host: Right, there’s a lot of — a lot of weird stuff happening right now. Yeah.

Host: So, what’s next?

35:23 What’s Next for Earendil

Host: Like, what do you want to do now? What are you currently working on? Stabilizing this thing? Or —

Mario: Yeah, I’m curious too.

Armin: Mostly dogfooding. Building a bunch of stuff internally, then deciding which parts to release.

Host: I mean, Armin, do you want to talk about this?

Armin: Yeah, I mean, I think you can roughly divide Earendil the company into two parts. One is — we’ve already built some products, so the products we’re building, ideally over the long term, would be enjoyable even for non-engineers. Not — not a coding agent.

Mario: Right.

Armin: So not a coding agent. We don’t want to be a coding agent company next year. At the same time, we’re also building infrastructure components that people can ideally use to build their own agents. So those are the two things we’re currently working on. And way back, before Pi, we built a product called Lefos — it — well, you’d call it a personal agent today, but it was really an agent sitting behind a plain-text communication

Armin: interface — in our case, email. We really loved that concept — a small entity sitting by your side, helping you solve problems, without imposing itself on the people outside. It’s not the “hey, help me book an appointment, I have an appointment” kind that goes off and harasses some real human with AI slop. It’s a useful assistant sitting next to you. Obviously there’s already a lot in this space.

37:02 Personal Agents and the Security Mess

Armin: But at the same time, we really don’t like anything out there in this space. They quickly turn into some kind of monstrous hybrid. So, just like Pi Durable, I think personal agents are extremely dangerous. I mean, the proposition is: you already have a bunch of people trying to hack you, and then you also have millions of people running GrokBot, logging everything into that machine. I think that’s a disaster — all of these companies are a disaster.

Mario: Actually, I’ve kind of always been surprised there hasn’t been more fallout. I mean, I’m enjoying it, man — it’s super fun. Actually, just five minutes before we started talking, I saw a post on Twitter from someone saying “my GrokBot sent all my personal financial information to my company’s Slack, and everyone can see it.” And Elon said if that happened, he’d pay to make it right for them, but he didn’t. Haha.

Host: Hmm.

Mario: Right. I like personal agents. I think — I’ve got a DeepSeek running at home right now.

Mario: I used to do this with Codex, but it was terribly slow. Like thirty-five — let me check — an average of thirty-three tokens per second, running through the Codex subscription. Absolutely terrible. So I run DeepSeek locally now. These open-weight models are good enough. They can scan about fifteen different emails in my inbox, convert them into Linear entries, and handle all that. But it’s still very messy, not particularly deterministic. Did you see Jeff’s whole thing?

Mario: I think — what really makes sense to me is: there are these — yeah, semi — what are they called again?

Armin: When I’m not the MCP lead, I’m Earendil’s chief chef.

Mario: He said if Armin could, he’d marry Jeff. He loves Jeff.

Armin: Merge those two together — I love chess.

Mario: That makes sense — merge the two together so you have one doing the categorization for the other. I think when they have to handle a lot of —

Links

Original video: https://www.youtube.com/watch?v=ja_7AF54OtE

Channel: 0xSero (Mario Zechner, Armin Ronacher)

This article is the second of three translations from the Pi Durable interview, with three parts in total.

#AgentEngineering #Durability