Loa/Arbor
English
  • English
  • 简体中文
  • 日本語
  • Español

Quality

Did that change make it worse? Find out on a real conversation.

Arbor saves every step of every conversation. Copy one at the step where it went wrong, change a prompt, a model or a tool, run it again, and read the two side by side with what each cost.

Example. arbor session fork --session s-1 --at 42 --replay --prompt style=./brief.md

Fix a bad reply, and check the fix

SpeedPlaying at 0.5× the speed it was recorded. Pauses longer than 2 seconds are cut to 2.
A real Claude Code session (Sonnet), unedited, on the support desk built on the Examples page. It finds a conversation where the desk gave no next step, deploys changed instructions as revision 4 without moving latest, forks the bad conversation onto it and runs arbor session compare. Two commands the allowlist refused are on screen too. The cost is $0 on both sides because the local model has no price, so the agent compares tokens instead. Recorded on Arbor's main branch at fb647c60.

Testing an agent

Without a record of every step

  • A user reports a bad answer. You have a log line and a guess.
  • You edit the prompt and try it on three inputs you invented.
  • Nobody can say if last week's change made the refund flow worse.
  • To rerun a conversation you rerun its tools, and the email goes out twice.

With Arbor's ledger

  • The bad conversation is already saved, step by step. It is the test case.
  • You copy it at the step that went wrong, change one thing, and run on from there.
  • The two runs are compared step by step, with each side's spend and the first step where they differ.
  • For each kind of call you choose who answers, so a tool that sent an email in the original does not send it again.

Any conversation is a test

A fork is a copy of a session up to a position that carries on from there with the change you choose. The original stays as it was.

Copy at the step that went wrong

arbor session fork --session s-1 --at 42 copies the session up to position 42. The copied steps are not run again, so they cost nothing. On a node not in --dev mode, a fork needs --replay. A replay fork is kept out of the session list and archived when it ends.

Change one thing

A prompt (--prompt id=file), a whole overlay of settings (--overlay), or the revision (--revision, or --label for one a label names). The fork runs on from the copy with that change.

Choose who answers each call

--intercept picks, per kind of call (the model, a tool, a module, an outside host), one of live, recorded, stub, generate, fuzz or fault. With none given, tools answer from the original's record and the model answers live.

Compare, with the cost

arbor session compare --session s-1 --with s-2 aligns the two sides' steps by what each step commits (messages consumed, calls made, messages sent, state left) and prints each side's spend and the first step where they differ.

Replay a session from the record

arbor session replay --session s-1 --revision 5 answers the calls the record holds from the record, and they cost nothing. After the first request the record does not hold, the replay goes on live, unless --strict stops it there. That is where a changed prompt first changes what the model is asked.

Forks and replays in the workflow

The loop

  1. Probe a live session

    arbor quality probe add --session s-1 --at 42 --set prompt.style='{"text":"brief"}'

    A probe forks the session at that position under your change, runs the copy and grades it by every eval whose matcher fires. `--base` runs the unchanged revision beside it, so you can tell the change's effect from the model's own variation.

  2. Keep the moment as a case

    arbor quality probe keep <probe> --benchmark refunds

    A case is a kept slice of a session. It carries the saved steps before the slice, which are never run again. Or formalize the probe as an experiment with `arbor quality probe formalize`.

  3. Run an experiment

    arbor quality experiment candidate add <experiment> --set prompt.style='{"text":"terse"}'
    arbor quality experiment gather set <experiment> --eval refunds
    arbor quality experiment gather run <experiment>
    arbor quality experiment run <experiment>
    

    An experiment has a base revision, numbered candidates (each a change laid over the base), a gather that finds more cases by an eval's condition in real sessions, and the evals that decide. A run prints one row per candidate, with its score and its cost.

  4. Hold some cases out

    arbor quality experiment add --question "..." --case <case id> --holdout 0.3
    arbor quality experiment case hold <experiment> <case id>
    arbor quality experiment holdout <experiment>
    

    `--holdout <share>` on `experiment add` holds out a share of the cases, and `experiment case hold` holds out one more. Held-out cases are not run while you iterate. `experiment holdout` grades them once, so you cannot tune until it agrees with you.

  5. Conclude

    arbor quality experiment conclude <experiment> --promote c2

    Promoting stages the winning change into the draft. It publishes nothing, so a person still reviews and publishes.

  6. Run a benchmark on each new revision

    arbor quality benchmark run refunds r3 r4 r5

    A benchmark is a standing set of cases. Run it against the revisions you name. It exits non-zero when a replay could not run, so it fits a CI step. `benchmark history` and `benchmark heatmap` show the score by revision against two bands, good and acceptable.

  7. Grade live traffic

    arbor quality monitor set web-refunds --evals refunds --trigger after:24h --sample 0.25 --ceiling 40 --ingress web

    A monitor runs evals over a sample of live sessions. A filter picks the sessions, the sample says how many, and the trigger says when to grade. The ceiling caps grading at forty an hour.

Every `arbor quality` command, in the workflow

How a conversation gets a grade

An eval is a question you can ask of any conversation, live or copied. When this happens, was it good?

A condition, not a step number

An eval's matcher finds the moments to grade: an actor of a definition ran, a tool was called, a message arrived, a label was set, the session closed. You do not point at step 42. You say what to look for, and every session that has it can be found with arbor quality eval matches.

Checks first, then one judge

An eval has any number of checks, which are plain assertions and cost nothing, and at most one judge: an agent with a rubric. A failed check scores 0. Otherwise the judge's score is the grade. With no judge, the grade is 1.

Every grade is one number from 0 to 1

So grades add up. A mean and a count, by revision, by arm or by time, mean the same whether the grades came from live sessions or from copies. arbor quality eval grades --by revision shows it.

Judging costs money, so it is counted

An eval with a judge costs one judge call per match. Every grade records what its judge cost. A monitor's sample and hourly ceiling keep the bill in bounds, and what was matched, graded, sampled out and clipped is counted, not guessed.

Arbor can play the user

In a copy, an agent can sit in the participant's seat, to see how a change holds up over a whole conversation and not one reply. You turn it on with a case that has a simulated feed, which needs a persona and a goal.

How the commands are named

What to know before you rely on it

A copy is not a replay of the model

The model answers live in a plain fork, and it can say something different each time. That is why a probe can run the base beside the change, and why an experiment can repeat each run.

A benchmark is run by you

A benchmark can be marked to run on every new revision (--every-revision), but the run is yours to start, from CI or by hand. Nothing triggers it for you.

A low grade does not fail a run

A run exits 1 when a replay could not run. A low score is a result you read in the history, not a failure.

The console pages are still being built

The commands exist today. The console's quality pages are built against them and are not finished.

Arbor holds itself to the same standard

4,172
mutants in the first sweep, over ten kernel and runtime files. Each is a small break in the code that some test should catch.
74%
of the 3,310 mutants that built and that a test covered were caught (2,450 caught, 860 survived). The survivors and the 256 uncovered mutants were sorted by hand, and tests were written for the ones that guard commits, leases, retry and park rules.
121
assumptions about the first eval design, since replaced, each checked against the real code. Six turned out to be bugs, and are fixed.

How Arbor is tested, and what it found

How the same ideas test Arbor's own runtime

The runtime is held to a written standard of its own, laid out in docs/quality-report.md and docs/2026-11-12-plan-quality.md.

For each part of the runtime that has been through it, three steps are done. The part’s states, events and waits are written down from the code. Each decision it makes is made visible as an event. Then an explorer drives it through every ordering of its events, with faults injected, and checks each against the written table. A combination with no entry in the table fails. An ordering that breaks its row is reported and never fixed in the same change.

%% The three steps each part of the runtime goes through
flowchart TB
  table["1 Write down states,<br/>events and waits<br/>from the code"] --> visible["2 Make each decision<br/>visible as an event"]
  visible --> explorer["3 An explorer drives every ordering<br/>of events, with faults injected"]
  explorer --> check{"Matches the<br/>written table?"}
  check -- "yes" --> done["Part passes"]
  check -- "no entry, or breaks its row" --> fail["Reported, not fixed<br/>in the same change"]

Other checks run beside that:

  • Mutation testing breaks the code in small ways, one at a time, and checks that a test fails for each break. The first full sweep is the number above. Four reports record the triage of its survivors (docs/2026-11-12-report-mutation-triage-runtime.md and the three of 15 November).
  • The race detector runs under the Go suite on every pull request.
  • Stress and soak runs look for wrong answers: lost messages, two turns of one actor committing, a parked actor waking. arbor soak drives scripted conversations through a running node.

It is not finished. Only the wake path has been through all three steps, by the quality report. Held calls and reading while writing are partly done: held calls have the written table and their fixes have tests, but their decisions are not yet visible and no explorer has run; reading while writing has the table and visible decisions, and no explorer yet. Ending a session has not been begun. The sweep covered ten files, not the whole runtime. Benchmarks shows what the load and stress tests found, and History lists every defect, with the open ones marked.