# Architecture · Loa Arbor

This page is for the people who run Arbor nodes. It explains the pieces, what a node does with a message, how a crashed turn is taken over, how far one Postgres goes, and where a tenant lives.

## Kill a node mid-turn

Terminal recording (https://arbor.getloa.app/casts/04-kill-a-node-mid-turn.cast): A terminal: two Arbor nodes share one Postgres; one is killed with kill -9 in the middle of an agent's turn, and the other takes the turn over and answers.

A real run, unedited. A script drives the `arbor` CLI, and prints itself first. Two nodes share one Postgres and one Redis, with 10-second leases. A bookshop's front agent gets three questions, and the node running its turn is killed with `kill -9` just after the turn saved its first `spawn`. When the lease lapses, the other node takes the turn over (`turn_crashed … lease_lost, recorded resume`). It keeps the researcher already started, starts the other two, and answers. Nothing failed. Two quirks show: after the takeover the agent started the remaining researchers one model call at a time, not together, and after answering it called the model once more for nothing. Recorded on Arbor's main branch at fb647c60.

Image: The console's Ledger for the bookshop session after the kill: steps 18 to 33, saved on the surviving node, where the researchers report, close, and the front agent replies.

The same run in the console: the session's ledger after the takeover. Every step the killed node saved is still there; the surviving node carries on from the next one.

## The pieces

Arbor is one program. A running copy of it is a **node**: one `arbor` process. A node does the work. It keeps nothing in memory that a conversation needs once a piece of work is saved, so you can start more nodes, stop some, or lose one, and no conversation depends on any single node.

The same binary runs in two profiles.

- **Local.** `arbor serve` runs one node on a SQLite file. It needs no Postgres and no Redis. It keeps sessions, actors, ledgers, bundles and settings in `arbor.db` and keeps artifacts beside it. This is the profile for development, tests and small deployments.
- **Cluster.** `arbor node`, run as many times as you need, on Postgres and Redis. Every node shares the same stores.

Moving from one shape to the other changes flags, not application code.

A node uses up to three kinds of store.

| Store | What it is | What losing it means |
|---|---|---|
| Coordination | Postgres in a cluster, SQLite on one node. Holds every durable fact: sessions, actors, their ledgers, messages, state, the meter. | Everything durable is here, so losing it loses the conversations. |
| Ephemeral | Redis in a cluster, in memory on one node. Holds live text while a reply is being written, and short-lived working data. | Nothing is lost from any conversation. Only live text in flight is affected. The saved record is in the coordination store. |
| Artifacts | A blob store: a local directory, or S3 with `--artifacts s3://bucket/prefix`. Holds files and other large objects. | Files are lost. |

You choose them with `--store memory | sqlite[:path] | postgres://…` and `--ephemeral memory | redis:<addr>`. `--in-memory` picks the in-memory pair, for tests.

```mermaid
%% Nodes, the three stores, and the product calling in
flowchart LR
  product(["Your product<br/>browser, backend, webhook"]) -->|"HTTP API"| nodeA["Node A<br/>arbor node"]
  product -->|"HTTP API"| nodeB["Node B<br/>arbor node"]
  nodeA --> pg[("Coordination<br/>Postgres")]
  nodeB --> pg
  nodeA --> redis[("Ephemeral<br/>Redis")]
  nodeB --> redis
  nodeA --> blob[("Artifacts<br/>blob store")]
  nodeB --> blob
```

Your product talks to Arbor over its HTTP API. Any node can answer any call. This page covers the running system. The API itself is on the Reference page.

## What a node does with a message

Arbor runs conversations as **actors**. An actor is a unit with an inbox for messages, a ledger that records what it has done, and some code that handles a batch of messages. An agent is an actor whose code is a loop around a model call.

Three facts about how this runs matter to an operator.

1. **An actor is a row.** An actor that is waiting has no process, no thread and no memory on any node. It costs storage. A conversation that waits on a person for a week costs the same as one that waits a second.
2. **Nothing runs between turns.** When messages arrive for an actor, some node picks the actor up, runs a **turn**, and lets go. A turn is the work for one batch of messages.
3. **One transaction per step.** A turn is a series of **steps**. A step is one model call, one tool call, one call to another system, or the end of the turn. Each step ends in a single write that saves everything the step did: the messages it consumed, the result of the call it made, the messages it sent and the actor's new state. All of it is saved, or none of it.

The third fact is what makes crash recovery possible. In Postgres, one stored procedure does that write in one transaction. It takes the session's row first, so the commits of one session happen one after another and the session's record has one order.

Posting a message returns at once. The turn runs on whichever node is free, and carries on if the caller goes away.

## Leases and crash recovery

Two nodes must never run the same actor at once. Arbor prevents this with a lease.

### The lease and the epoch

A node that wants to run an actor **claims** it. The claim takes a **lease**: the actor belongs to that node for a time. The default `--lease-ttl` is 30 seconds. The node renews the lease every third of that time while it is working.

Each claim also raises the actor's **epoch**, a counter. Every step a node writes carries the epoch it claimed at. If the actor has since been claimed at a higher epoch, the store refuses the write. That refusal is what stops an old node's late write. A node that was only slow, and wakes up after another node took over, cannot write over the new owner's work.

In Postgres a node finds work with a query that skips rows another node is already taking (`SKIP LOCKED`). Nodes learn that work is waiting from a Postgres notification sent inside the commit that created the work. Nodes also poll, once a second by default, in case a notification is lost. For work waiting to be taken, a lost notification costs at most one poll. An actor already resident and idle on a node is different: a notification it loses waits for the rest of its idle hold, up to five minutes, which is an open issue (Benchmarks has the run that found it).

### When a node dies mid-turn

```mermaid
%% Node A dies after step 2; node B takes the turn over
sequenceDiagram
  participant A as Node A
  participant S as Postgres
  participant B as Node B
  A->>S: claim actor, lease, epoch 1
  A->>S: commit step 1 (model call)
  A->>S: commit step 2 (tool call)
  Note over A: node A dies in the middle of step 3
  Note over S: the lease is not renewed and lapses
  B->>S: claim actor, epoch 2
  S-->>B: the turn is unfinished: steps 1 and 2
  B->>S: commit step 3, epoch 2
  B->>S: commit final step, epoch 2
  A-xS: late write at epoch 1 (if A was only slow)
  S-->>A: refused, stale epoch
```

The steps before the crash are in the ledger. Node B reads them and carries on from the last one. The crash costs at most the one step that was in flight.

- **An agent** reads back the model answers and tool results already saved, and makes the next call. A finished call is not made twice.
- **A plain actor** (ordinary code, no model) runs again from the top over the same messages. Each call it already made returns its saved result without running, so it reaches the crash point with the same values.

The turn counts as unfinished until its last step is saved. So another node claims it even if no new message is waiting.

### One limit

Arbor cannot know whether a tool that was running at the moment of the crash had finished. What happens next depends on how the tool is marked.

- A tool marked read-only or safe to repeat runs again, under the same call id.
- Any other tool, destructive ones included, is not run again, and the agent is told the outcome is unknown.
- A tool's author can choose otherwise for it: wait for a person to decide (`ask`), or stop the actor (`park`).

Making a repeat harmless is the job of the tool and the system behind it. Arbor hands every tool its call id as an idempotency key for that purpose.

### Several nodes on one conversation

A lease belongs to one actor, not to a conversation. A session usually holds several actors: an assistant, a subagent it started, perhaps a second root. Each is claimed on its own. So one conversation can have its actors running on several nodes at the same moment. The steps of each single actor stay in order, because only the lease holder can write them. The commits of the whole session still queue on the session's row, which keeps its order and its spend exact.

## How far one deployment goes

Nodes are built to scale out; one and two nodes have been measured. A Postgres primary does not scale out. Every commit of a deployment goes to its primary, so the primary is the ceiling. Here is what has been measured.

### The latest figure

On 16 November 2026, a load test measured one node on Postgres with a real Valkey server (a Redis-compatible store) as the ephemeral store, and 200 sessions at once. The model was a scripted one, and the machine was a development machine, not a server. The pool was 32 connections. Two runs gave **1,086** and **1,093 turns a second**. A node spent about 12 ms on a turn. This is one node, so it says nothing yet about adding nodes.

An older figure, about 730 turns a second at 50 sessions and 180 at 200, came from a test that used a fake in-process Redis. The November test showed that the fake set that limit, not Arbor. Use the 1,086 figure.

### The commit on its own

On 5 October 2026, the commit procedure alone was measured on an Apple M5 Pro laptop against Postgres 17.5. This is not a load test of a node.

| Commit test | Commits a second | Median commit |
|---|---|---|
| 1 session, 1 actor | 1,196 | 0.60 ms |
| 1 session, 4 actors | 1,437 | 2.36 ms |
| 1 session, 16 actors | 1,324 | 11.56 ms |
| 16 sessions, 1 actor each | 10,414 | 1.18 ms |

Read the table this way. One very busy session tops out around 1,300 to 1,500 commits a second, because its actors queue on the session's row. Many sessions together go much higher. A turn with a model or a tool in it commits a step every few seconds, so ten thousand commits a second is far more than ten thousand busy conversations. A conversation that waits on a person costs only storage, so how many can be open at once is a question of disk.

### Not measured yet

- A managed Postgres with synchronous replicas, where each commit is slower.
- More than two nodes.

The Benchmarks page has the full setup behind each number.

## Placement: where a tenant lives

One primary is the limit of one database. What would let a deployment grow past it is placement. This section says what is built.

### The placer and the placed id

A tenant in Arbor is a **partition**: the key an app declares for a session, such as a customer account or a campaign. Everything that must stay together stays inside one partition, so the partition is the **unit of placement**.

When a session opens, a function called the **placer** takes the app and the partition and answers a **placement**: a short name for the tenant. The default placer answers the app's name, so every app is a tenant from the start. A deployment chooses its placer once, at boot.

The placement is written into every id the kernel mints for the session. The session's id, its actors' ids and its bindings' ids all take the form `<placement>.<uuid>`. Every row in the database names one of those ids, so every row already knows its placement. A request that names a session or an actor carries the placement with it, so a store that routes by placement needs no lookup. The placement is a name, not a location. Moving a tenant to another database would change a table and leave every id valid.

### A placer of your own

A custom placer is a Go function. You hand it to the node in a program of your own that imports Arbor as a library:

```go
cli.NewRoot(arbor.WithPlacer(yours))
```

That builds the same binary, with no fork to keep up to date. A placer must be a pure function of the app and the partition. It cannot depend on the session, which does not exist yet, or on who is calling.

### Reads across tenants

A tenant's own requests touch one database. A read that spans tenants, such as an operator's list of every session or stats across apps, becomes one query per database, merged by the store.

### What is built and what is not

The Arbor source says this about the state of placement:

> The placer, the placed ids and the placement on every row are in Arbor. The store that routes by placement exists, and its job today is to keep the abstraction honest: it runs the whole contract suite over two in-memory databases, with sessions landing on both, and the runtime's fault tests on top. The same store over several Postgres databases, and the table that says which tenant lives on which, are not built; they wait for a deployment that needs them. Until then every deployment is one Postgres database, and that is what the numbers above measure.

So the design is in place and has been proved over in-memory databases. Running one deployment across several Postgres databases is not built yet. Today a deployment is one Postgres primary.

## Operating it

### TLS

A node serves HTTPS whenever it has a certificate. Give it one with `--tls-cert` and `--tls-key`, or put mkcert's files in `certs/` (`certs/localhost.pem` and `certs/localhost-key.pem`). Plain HTTP is allowed on loopback only. A node that listens on other interfaces without a certificate refuses to start unless you pass `--insecure-http`. HTTP/2 is used under TLS, so one connection carries every live view a console page opens.

### Health

The health probe is `GET /healthz`. The API is under `/v1/`, and `/` redirects to the console.

### The console

Every node serves the operator console at `/ui/`, beside the API. It shows sessions, each actor's status, epoch and residency, the ledger as it happens, the cost tree and traces. It has fleet pages for nodes, activity, and configuration with drift. The same node data is at `GET /v1/admin/nodes`, which also lists each node's version. On the local profile, open `http://127.0.0.1:8080/ui/` with nothing to configure. Elsewhere, paste an operator token once.

### OpenTelemetry

A node's metrics are at `GET /v1/admin/metrics`. With `--otlp`, the node also exports traces, logs and metrics to an OTLP collector. `--otlp` takes `host:port` for a plain-HTTP collector, or a base URL. A turn is one trace, and spans, logs and metrics carry the same labels: session, actor, epoch and step.

### The schema

On Postgres, run `arbor admin migrate` before the first `arbor node`. It applies the schema and stored procedures embedded in the binary to the Postgres named by `ARBOR_POSTGRES_URL`. Arbor is pre-release, and stored data does not yet survive a schema change.

- [Benchmarks](https://arbor.getloa.app/en/../benchmarks/): Every measurement behind the numbers on this page, with its setup and date.
- [Lifecycle](https://arbor.getloa.app/en/../lifecycle/): How an app is built, released and pinned to the sessions that started on it.
- [Reference](https://arbor.getloa.app/en/../reference/): The API, the commands and the settings.

Loa Arbor. © Richard Clayton. All rights reserved. Intended licence: FSL-1.1-MIT, at launch. Coming soon: the download opens when the repository goes public.
