# App lifecycle · Loa Arbor

An Arbor app changes in small, named steps, and nothing you publish ever reaches into a conversation that is already running. This page explains the pieces, then follows one app through a first publish, a settings change, new code, a gradual move and a rollback.

## Done by a coding agent

Terminal recording (https://arbor.getloa.app/casts/03-settings-draft-and-a-pinned-revision.cast): Claude Code, in a terminal, staging an Arbor app's new model role and instructions as settings, reviewing the draft and publishing a new revision while an older conversation stays on its own.

A real Claude Code session (Sonnet), unedited except for the machine's username, which is replaced. It picks up the support desk built on the Examples page. The agent adds a model role and a prompt in code and deploys them as revision 5, then opens a conversation on it. Then it stages a new model and new instructions with `arbor settings set`, reviews the draft and publishes revision 7. The conversation opened before stays on revision 5, and a new one opens on 7. It goes wrong on screen too: two commands the allowlist refused, and a model the local node can't reach, so the last chat on revision 7 fails with "model not found". The list of revisions at the very end shows only 1 to 3 because the recording script asked for those three; the agent's own listings earlier show 5 and 7. Recorded on Arbor's main branch at fb647c60.

An app that talks to people for weeks cannot be edited like a script. Someone is in the middle of a conversation when you change a prompt. Half your users should try a new model first. The new model turns out to be worse, and you want the old one back before lunch. Arbor's answer to all three is the same: **things that run are never edited; you publish a new one and move a pointer to it.**

This page explains how that works first, in plain words, and then runs one small app through it. The commands are the "how" and come second. [The dev workflow](../workflow/) page lists them all in order.

## The idea in one paragraph

An app is code plus settings. When you publish, Arbor takes a snapshot of both and calls it a **revision**. A conversation (a **session**) opens on one revision and stays on it until it ends. Your product does not name a revision; it opens conversations through an **ingress**, a named address that says which revision new conversations get. To change what users see, you publish a new revision and move the ingress. To undo it, you move the ingress back. The old revision was never touched.

## The pieces, and why each exists

### The app, its code, and its settings

An **app** is one bundle of TypeScript (written with `@arbor/sdk`) that runs in an environment. Two different things make it behave as it does, and the model keeps them apart on purpose.

- **The code** is the bundle: actors, tools, the prompts it ships in `prompts/`. You change it with a **release**, which is a build tagged with a version that you chose (the `version` in `package.json`, such as `0.1.0`). A release never changes. Publishing different sources under a version that is already released is refused, and the refusal names the version to use.
- **The settings** are everything you may want to change without a code release: which model an agent uses, a prompt's text, a tool's description, a per-session budget, a retention period. They are one document of flat keys such as `actor.front.model.model`, `prompt.style`, `tool.lookup.description` and `app.budget.session_micros`. `arbor settings schema` lists every key, with its type, default and description.

Why split them? Code review and a version bump are the right ceremony for logic. They are the wrong ceremony for "make the support agent a little shorter" or "cap a session at 25 cents". Settings change often and are safe to change; code changes rarely and carries risk.

Settings come from three places that are merged in a fixed order, later over earlier:

| Settings document | Where it lives | Who writes it | Travels with a deploy? |
|---|---|---|---|
| Defaults | `defaults` in `arbor.yaml` | the developer | yes, to every environment |
| Environment values | the environment's node | the environment's admin | never; not in the repository |
| Overrides | the environment's node | an operator, by publishing a draft | survive deploys; `arbor pull` can turn them into defaults |

Some keys have one home only. A model's configuration (`model.<role>.*`, `actor.<name>.model`), an HTTP endpoint's `url`, `credential` and `headers`, and a few others belong to the environment: a defaults document that sets one is refused. That is deliberate. The code says "I need a model called `smart`"; each environment says which provider and model that is, because a test node and production rarely agree.

### The revision: what a session pins

A **revision** (`r1`, `r2`, ...) is one release plus the settings as they were at the moment of publishing. Numbers count up per app, per environment, and are never reused. A revision never changes after it is made.

A session pins its revision when it opens, and then keeps it for life. At that moment it also resolves what the settings point at and pins the result: a prompt's text, the provider and model an actor will use. So a running conversation reads its own pinned copy and nothing else.

![The console's Revisions page for a support desk: r7 is published and serves `latest`, while sessions opened earlier keep running on r5 and r1.](./console-revisions.png)

This is the guarantee the rest of the page leans on: **publishing a new version never changes a conversation in progress.** You can publish, move labels, move ingresses and roll back while sessions run, and none of those sessions notices.

Two exceptions:

- A prompt can point at a movable tag in the prompt store (`prompt.style: store:style@stable`). The text is resolved when each session opens, so two sessions of one revision can pin different texts if you retag in between. Once a session has opened, its text is fixed.
- A few settings are read live on purpose, because a running session should follow them: `schedule.<name>.*` (pause, resume, a new spec), `app.retention.*` (when to archive and purge) and the coding settings (`coding.*`). Secrets are by reference (see below), so a rotated credential reaches running sessions at once. Everything else is pinned.

### The draft: why settings changes are staged

If every `settings set` took effect the moment you typed it, two edits would be two live changes, a typo would reach users, and nobody could review. So settings changes go through a **draft**: the app's one pending set of changes.

1. **Stage** keys into the draft. Nothing is published.
2. **Review** it: each change as `key: from → to`, what **blocks** the publish (a required value missing, a prompt pointer that does not resolve) and what only **warns**.
3. **Publish** it. The whole draft becomes one new revision, and the `latest` label moves to it.

The draft protects you from three things: a half-finished change going live, a change nobody looked at, and two people overwriting each other. If someone published while you were drafting, your publish fails with `conflict.draft_stale` instead of silently winning, and you review again. A draft left pending is a normal place to stop: the agent or developer says what it changes, and a person decides.

```mermaid
%% How a settings change goes from a draft to a new revision
flowchart LR
  stage["Stage keys<br/>nothing is live"] --> review["Review<br/>blocks and warnings"]
  review --> publish["Publish"]
  publish --> rev["New revision<br/>latest moves to it"]
  publish -. "someone published first" .-> stale["conflict.draft_stale<br/>review again"]
  stale -.-> review
```

### Labels: names for revisions

A **label** is a movable name on a revision, like a git tag: `latest`, `stable`, `canary`, any name of up to 32 lowercase letters, digits, `_` or `-` (but not `rN`). Publishing moves `latest` to the new revision. Other labels move only when you say so. A label has no effect on traffic by itself; it is a name that an ingress can follow.

### The environment: where a copy of the app lives

An **environment** is a node (or a fleet) that the project knows by name, such as `local`, `play` or `prod`. Each has its own revision numbers, its own environment values, its own overrides, its own labels and ingresses. Revision 4 on `play` and revision 4 on `prod` are unrelated. That separation is the point: you try things on `play`, and `prod` moves only when you say so.

The project lists its environments in `arbor.yaml` (a name and a host), and keeps each one's token under your home directory, never in the repository. Every command takes `--env <name>` and defaults to `local`.

### The ingress: how your product reaches a revision

Your product should not hard-code `r7`. It opens a session through an **ingress**, a named address that holds a target:

- a **label**: new sessions get whichever revision the label names now;
- a fixed **revision**: new sessions always get that one;
- a **split**: weights that sum to 100, across revisions and labels (for example `stable=90,canary=10`).

Every app has an ingress called `latest` that follows the `latest` label. You add your own, such as `prod`. An ingress also has a kind (`http`, or `a2a` for an app that declares an A2A composition; an `a2a` ingress requires a bearer token and refuses an anonymous caller, so a public, anonymous surface is an `http` ingress), may be private (only operators and services can open through it), and may push each outbox entry of its root to a webhook. Which roots the outside can reach is the bundle's `session.exposed` list, not the ingress's.

Only **new** sessions follow an ingress. Moving one changes nothing for a conversation that has already started. Every move is logged: `arbor ingress log` says who moved it where, and when.

### Putting it together

```mermaid
%% How code and settings become what a new conversation runs
flowchart TB
  code["Your code<br/>package.json 0.1.0"] -- "deploy" --> release["Release 0.1.0<br/>never changes"]
  settings["Settings<br/>defaults + environment + overrides"] -- "draft publish or deploy" --> revision
  release --> revision["Revisions r1, r2, r3 ...<br/>a release plus settings, never change"]
  revision -- "named by" --> labels["Labels: latest, stable, canary<br/>move when you say"]
  labels -- "followed by" --> ingress["Ingresses: latest, prod<br/>move when you say"]
  ingress -- "new sessions open through" --> sessions["Sessions<br/>each pinned to one revision for life"]
```

Everything above the sessions can move. Nothing below the line "revision" is edited.

### Who changes what, and how

| Thing | Where it lives | Changed by | How |
|---|---|---|---|
| Code | a release, built from `app.ts` and `prompts/` | the developer | edit, raise `package.json`'s version, `arbor deploy --env <env>` |
| Defaults | `arbor.yaml` | the developer | edit the file; they ship with the next deploy |
| Environment values (model provider and model, endpoints) | the environment | the environment's admin | `arbor settings set model.smart.model=<model> --env <env>`, then publish |
| Overrides | the environment | an operator | `arbor settings set ...`, `arbor draft publish --env <env>` |
| A revision | the environment's catalog | nobody; it is made, not edited | `arbor deploy` or `arbor draft publish` makes the next one |
| A label | the environment | an operator | `arbor label set <label> --revision <n> --env <env>` |
| An ingress | the environment | an operator | `arbor ingress set <name> --label <l>` or `--revision <n>` or `--split ...` |
| A session's revision | the session | nobody | chosen once, when it opens |
| Secrets | the environment's secret store | an admin | by reference only (below) |

### Secrets and variables are references

A credential is never a value in a settings document or in `arbor.yaml`. It is written as a reference, `secret:<ref>` or `${NAME}`, and the node resolves it at the moment of use. A literal written where a secret belongs is shown back as `[redacted]` and is refused in a project's defaults. An admin puts the value in the environment's secret store with `arbor admin secret set <scope>/<name>`, from a file or an environment variable, never as an argument, and it is never read back. Because the reference is pinned and the value is looked up live, rotating a secret needs no publish. A plain variable (`var.GREETING`) is an ordinary setting that the bundle declares and an environment fills in.

### Overlays: a trial, not a release

An **overlay** is a sparse patch of the same flat keys, laid over a revision for one question: what if this prompt were shorter? The tools that ask "what if" use it: `arbor quality probe`, `arbor quality experiment` and `arbor session fork --prompt style=./brief.md`. A session can also be given a value for a setting that allows it, at open: `arbor session open --set app.caps=...` (the key `app.caps` is the first that does). An overlay is never a live release. The only way a winning candidate reaches users is `arbor quality experiment conclude --promote`, which **stages** it into the draft, so the same review and publish stand in front of it.

### Rollouts: what is built

Moving traffic gradually is done by hand, with a split on an ingress. There is no automatic ramp that steps from 10 to 50 to 100 percent, and no rule that rolls back when a number gets worse. You choose the weights, watch the result (`arbor stats --app <app> --by revision`, or `--by ingress`), and move them again. The older design for gateway routes and a canary header is gone; the ingress replaced it.

## One app, from first publish to rollback

Here is a small support app, `helpdesk`, on an environment called `play`. The story has six steps. After each, a session that opened earlier is still on the revision it started on, and the table at the end shows the whole run at once.

### 1. The first publish

You have an app and want to ship it. The first publish makes a release and the first revision, and moves `latest` to it.

```sh
arbor deploy --env play --note "first deploy"
```

`deploy` sends the sources (the node builds them), releases the version in `package.json` (`0.1.0`), publishes revision 1 and moves `latest` to it. If the environment is missing something the app needs, such as a model for a role, the release is kept and the revision is refused, with one line per missing value and the command to fix it. That is the common first-time stop: set the value (it is a settings change, as in step 2), publish the draft, and you do not upload again.

Your product should not follow `latest`, which moves on every publish. Give it its own labelled address:

```sh
arbor label set stable --revision 1 --env play
arbor ingress set prod --label stable --env play --note "the product's door"
```

Now `prod` hands out whatever `stable` names. A conversation opens:

```sh
A=$(arbor session open --ingress prod --env play)
arbor session get --session "$A" --env play
```

`session get` prints the session's app, release and revision on its first line. This one says `r1`. Call it session A.

### 2. A settings change: drafted, reviewed, published

Sessions cost too much, and you want to cap each at 25 cents. That is a settings change, so it goes through a draft:

```sh
arbor settings set app.budget.session_micros=250000 --env play --note 'cap a session at 25 cents'
arbor draft review --env play
arbor draft publish --env play
```

`settings set` stages the key into the draft and prints the review, with the change as `app.budget.session_micros: unset → 250000`. Nothing is live yet. `draft review` lists the changes, the blocks and the warnings. `draft publish` makes **revision 2** (same release, `0.1.0`, new settings) and moves `latest` to it. The command prints the revision and `latest -> 2`.

Now check on session A:

```sh
arbor session get --session "$A" --env play
```

It still says `r1`, and it still has its old budget. Revision 2 exists, but nothing has pointed new sessions at it either: `stable` still names revision 1, so `prod` still hands out revision 1. Publishing and releasing to users are two separate steps, and the second is yours. When you have looked at it:

```sh
arbor label set stable --revision 2 --env play
```

New sessions through `prod` now get revision 2. Session B, opened after this, says `r2`. Session A carries on at `r1` until it ends.

### 3. New code

The support agent needs a new tool. That is code, so it needs a new release. Edit `app.ts`, then raise the version in `package.json` to `0.2.0`. The same version with different sources is refused (`conflict.version_taken`), and a change to the app's contract (its roots, roles, inputs and the like) must raise the major version, or the minor below 1.0 (`conflict.contract_changed`).

You do not want the new code to take over immediately, and `deploy` moves `latest` by default. So give it a different label:

```sh
arbor deploy --env play --label canary --note "adds the lookup tool"
```

This makes release `0.2.0` and **revision 3**, which carries the settings of revision 2 (the budget cap) with the new code. It puts the `canary` label on revision 3 and leaves `latest` and `stable` where they were. Nobody is on it yet.

### 4. Moving traffic to it

Send a tenth of new conversations through the new revision:

```sh
arbor ingress set prod --split stable=90,canary=10 --note ramp --env play
arbor ingress log prod --env play
```

`ingress set` creates or changes the ingress and keeps any flag you leave out. Each new session is placed by those weights. `arbor stats --app helpdesk --by revision --window 24h --env play` shows how each revision is doing. When you are happy, change the weights again, or send everything to it:

```sh
arbor ingress set prod --label canary --env play
```

Sessions that opened earlier stay where they are: A on revision 1, B on revision 2, and a canary session on revision 3.

```mermaid
%% Where new and existing sessions are after the ramp
flowchart LR
  prod["prod ingress<br/>stable=90, canary=10"] -- "90" --> r2["r2 (stable)"]
  prod -- "10" --> r3["r3 (canary)"]
  A["Session A"] -. "pinned" .-> r1["r1"]
  B["Session B"] -. "pinned" .-> r2
  C["A canary session"] -. "pinned" .-> r3
```

`ingress set` stages and publishes a one-change draft for itself, so it refuses with `conflict.draft_pending` while the app has a draft of its own waiting. Publish or discard that first.

### 5. Rolling back

The new tool has a bug. Rolling back is moving the pointer back, and nothing is rebuilt or deleted:

```sh
arbor ingress set prod --label stable --env play
```

New sessions through `prod` are on revision 2 again. The sessions already on revision 3 are not moved; they finish on the code they started on. (Arbor never moves a running session.) If you want revision 3 to stay unused, leave `canary` where it is; it is inert.

The same idea undoes a plain publish. If your product opens sessions through `latest` and a new revision misbehaves, move the label back:

```sh
arbor label set latest --revision 2 --env play
```

There is no default label for `label set`; you always name it. The revision's settings come back with it, because they were part of the revision.

### 6. Promote to production

Everything above ran on `play`. To ship the release you tested, without rebuilding anything:

```sh
arbor deploy --env prod --release 0.2.0 --from play
```

This takes the released sources of `0.2.0` from `play`, builds them on `prod`, and refuses with `conflict.digest_differs` if the build does not match the origin's. `prod` makes its own revision from its own settings (its own environment values and overrides), and you repeat the label and ingress moves there.

### The whole run

```mermaid
%% The six steps in order, with the pointer each one moves
flowchart TB
  s1["1 First publish<br/>r1 made, latest and stable on r1"] --> s2["2 Settings change<br/>r2 made, stable moved to r2"]
  s2 --> s3["3 New code<br/>r3 made, canary on r3"]
  s3 --> s4["4 Ramp<br/>prod splits 90 stable, 10 canary"]
  s4 --> s5["5 Rollback<br/>prod follows stable again"]
  s5 --> s6["6 Promote<br/>release 0.2.0 built on prod"]
```

| After step | Revisions | `stable` | `canary` | `prod` ingress | Session A | A session opened now |
|---|---|---|---|---|---|---|
| 1. First publish | r1 | r1 | none | follows `stable` | opens on r1 | r1 |
| 2. Settings published, then `stable` moved | r1, r2 | r2 | none | follows `stable` | still r1 | r2 |
| 3. New code, labelled `canary` | r1, r2, r3 | r2 | r3 | follows `stable` | still r1 | r2 |
| 4. Ramp | r1, r2, r3 | r2 | r3 | 90% `stable`, 10% `canary` | still r1 | r2 or r3 |
| 4b. Full move | r1, r2, r3 | r2 | r3 | follows `canary` | still r1 | r3 |
| 5. Rollback | r1, r2, r3 | r2 | r3 | follows `stable` | still r1 | r2 |

The one thing that never appears in the table is a revision changing. Only pointers move.

## What to remember

- A revision is code plus settings, frozen. A session pins one for life.
- Settings change through a draft: stage, review, publish. Code changes through a release with a version you choose.
- Publishing makes a revision. Labels name revisions. Ingresses decide which revision new sessions get. Each step is separate, so you can stop between them.
- Rollback is moving a label or an ingress back.
- Environments are separate copies; promote a release between them with `deploy --release ... --from ...`.
- Writes that go straight to live state are separate commands on purpose (`settings override`, `settings replace`, `label set`, `ingress set`, `deploy`), so a permission rule can tell them from staging.
- `settings override` is the odd one: it writes the overrides document but publishes nothing, so nothing sees it until the next revision is published (`arbor draft publish`, or `arbor deploy`).

- [Dev workflow](https://arbor.getloa.app/en/../workflow/): Every command of the build, run, test and ship loop, in order.
- [The building blocks](https://arbor.getloa.app/en/../blocks/): What goes inside the app whose life this page describes.
- [Reference](https://arbor.getloa.app/en/../reference/): Look a name up.

Loa Arbor. © Richard Clayton. All rights reserved. Intended licence: FSL-1.1-MIT, at launch. Coming soon: the download opens when the repository goes public.
