# Runs & statistics

Why every result carries n and a ci95, and how auto(ci=0.95, moe=0.03) sizes a run and stops it early.

A success rate is a claim about a binomial process: some number of episodes
ran, some succeeded. Strip away how many and how tight, and the number is
unfalsifiable. So every result carries its sample size and its interval —
`n` and `ci95` — a bare percentage is a bug, in the API and on this site.

## Reading a result

```text
n=612  success_rate=0.874  ci95=(0.846, 0.898)
failure_clusters: [grasp_slip: 41, perception_miss: 22, collision: 9]
robot_hours=20.4  environment_hours=20.4  cost_usd=3,812
artifacts: mcap[], video[], ground_truth[], report_pdf
```

Episode outcomes are binary, so success rates are binomial proportions. We
report Wilson score intervals because they behave correctly near 0 and 1 —
exactly where deployable policies live — where the textbook Wald interval
degenerates (an observed 1.0 gets a zero-width interval). Read
`ci95=(0.846, 0.898)` as: the observed rate is 0.874 (n=612), and rates
outside that interval are hard to square with the data at 95% confidence.

## Sizing a run with auto()

Pass an int to `episodes` if you already know your n.
`episodes="auto(ci=0.95, moe=0.03)"` solves for precision instead. It plans
conservatively from p=0.5, the worst case for binomial variance — n≈1,067
episodes for a ±3-point margin of error at 95% confidence — then sizes
sequentially: the Wilson interval is recomputed as episodes complete, and the
run stops as soon as the observed interval tightens under the target. The
canonical run stopped at n=612 with rate=0.874, ci95=(0.846, 0.898) — a
half-width of ≈0.026, under the 0.03 target and roughly 450 episodes short of
the plan. You pay for the precision you asked for, not for the worst case.

| Target moe | Planning n at p=0.5 |
| --- | --- |
| 0.05 (±5 pts) | ≈385 |
| 0.03 (±3 pts) | ≈1,068 |
| 0.02 (±2 pts) | ≈2,401 |

These are planning upper bounds — computed at the variance-maximizing p=0.5
and rounded up to whole episodes. Sequential stopping usually lands lower,
and the further the true rate sits from 0.5, the earlier the stop.

*Example: auto-sized run*

**Python**

```python
import roborama  # reads ROBORAMA_API_KEY from the environment

run = roborama.run(
    robot="g1-edu-pro@fw2.3",
    environment="kitchen-std@v1.2",
    task="pick_place@v1",
    episodes="auto(ci=0.95, moe=0.03)",
)
print(run.result())  # n=612  rate=0.874  ci95=(0.846, 0.898)
```

**TypeScript**

```typescript
import Roborama from "@roborama/sdk"; // reads ROBORAMA_API_KEY

const roborama = new Roborama();

const run = await roborama.runs.create({
  robot: "g1-edu-pro@fw2.3",
  environment: "kitchen-std@v1.2",
  task: "pick_place@v1",
  episodes: "auto(ci=0.95, moe=0.03)",
});

console.log(await run.result()); // n=612  rate=0.874  ci95=(0.846, 0.898)
```

**cURL**

```bash
curl https://api.roborama.com/v1/runs \
  -H "Authorization: Bearer $ROBORAMA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "kind": "run",
    "robot": "g1-edu-pro@fw2.3",
    "environment": "kitchen-std@v1.2",
    "task": "pick_place@v1",
    "episodes": "auto(ci=0.95, moe=0.03)"
  }'
```

**Agent (tool-use payload)**

```json
{
  "type": "tool_use",
  "name": "roborama_run",
  "input": {
    "robot": "g1-edu-pro@fw2.3",
    "environment": "kitchen-std@v1.2",
    "task": "pick_place@v1",
    "episodes": "auto(ci=0.95, moe=0.03)"
  }
}
```

## Separating two policies

Precision on one rate is half the job; the other half is telling two policies
apart. Run them independently and the difference pays for both runs' noise.
[compare()](/docs/primitives/compare/) with `paired=True` runs both policies
against the same initial conditions, episode pair by episode pair, so the
shared variance cancels — pairing roughly halves the n needed to separate
them. In the worked example, v4 beats v3 by +4.1 points (p=0.008, 95% CI on
the effect +1.1 to +7.1 points).

## Failure clusters

The result doesn't stop at a rate. Failures arrive clustered — in the
canonical run, `grasp_slip: 41, perception_miss: 22, collision: 9`, which
accounts for 72 of its 77 failures. Clusters need not partition all failures:
the five that fit no cluster stay unlabeled rather than force-fitted. Every
cluster member links its episode artifacts — video, MCAP, replay spec — so
"what broke" is a queue of replayable episodes, not a guess. The artifacts
themselves are specified in the [data contract](/docs/concepts/data-contract/).