Runs & statistics

Why every result carries n and a ci95, and how auto(ci=0.95, moe=0.03) sizes a run and stops it early.

A success rate is a claim about a binomial process: some number of episodes ran, some succeeded. Strip away how many and how tight, and the number is unfalsifiable. So every result carries its sample size and its interval — n and ci95 — a bare percentage is a bug, in the API and on this site.

Reading a result

Episode outcomes are binary, so success rates are binomial proportions. We report Wilson score intervals because they behave correctly near 0 and 1 — exactly where deployable policies live — where the textbook Wald interval degenerates (an observed 1.0 gets a zero-width interval). Read ci95=(0.846, 0.898) as: the observed rate is 0.874 (n=612), and rates outside that interval are hard to square with the data at 95% confidence.

Sizing a run with auto()

Pass an int to episodes if you already know your n. episodes="auto(ci=0.95, moe=0.03)" solves for precision instead. It plans conservatively from p=0.5, the worst case for binomial variance — n≈1,067 episodes for a ±3-point margin of error at 95% confidence — then sizes sequentially: the Wilson interval is recomputed as episodes complete, and the run stops as soon as the observed interval tightens under the target. The canonical run stopped at n=612 with rate=0.874, ci95=(0.846, 0.898) — a half-width of ≈0.026, under the 0.03 target and roughly 450 episodes short of the plan. You pay for the precision you asked for, not for the worst case.

These are planning upper bounds — computed at the variance-maximizing p=0.5 and rounded up to whole episodes. Sequential stopping usually lands lower, and the further the true rate sits from 0.5, the earlier the stop.

import roborama  # reads ROBORAMA_API_KEY from the environment

run = roborama.run(
    robot="g1-edu-pro@fw2.3",
    environment="kitchen-std@v1.2",
    task="pick_place@v1",
    episodes="auto(ci=0.95, moe=0.03)",
)
print(run.result())  # n=612  rate=0.874  ci95=(0.846, 0.898)

Separating two policies

Precision on one rate is half the job; the other half is telling two policies apart. Run them independently and the difference pays for both runs' noise. compare() with paired=True runs both policies against the same initial conditions, episode pair by episode pair, so the shared variance cancels — pairing roughly halves the n needed to separate them. In the worked example, v4 beats v3 by +4.1 points (p=0.008, 95% CI on the effect +1.1 to +7.1 points).

Failure clusters

The result doesn't stop at a rate. Failures arrive clustered — in the canonical run, grasp_slip: 41, perception_miss: 22, collision: 9, which accounts for 72 of its 77 failures. Clusters need not partition all failures: the five that fit no cluster stay unlabeled rather than force-fitted. Every cluster member links its episode artifacts — video, MCAP, replay spec — so "what broke" is a queue of replayable episodes, not a guess. The artifacts themselves are specified in the data contract.