# Roborama — full documentation

Generated 2026-09-10. Canonical URLs are listed per section; every section is also available as its own .md file (same path, .md extension).

---

<!-- source: https://roborama.com/docs/ -->

# Roborama documentation

Real robots as an API — evaluate policies on physical hardware and get back results with n and confidence intervals.

Roborama runs your robot policy on real, instrumented hardware and returns a
statistically defensible result: a success rate with its sample size and a
Wilson confidence interval, failure clusters, full episode artifacts, and a
citable verification report. Simulation screens; physical testing verifies.

## The mental model

```text
task (draft→pilot→frozen) → quote → run|eval|verify|matrix|threshold
                                → episodes (MCAP + video + GT)
                                        → result (n, rate, CI, clusters)
                                        → report (verification, citable)
                                        → webhook (gate your release)
```

Freeze the claim as a [Task Spec](/docs/concepts/tasks-and-method-transfer/),
quote a job, run a primitive — and everything that comes back is an
artifact, a statistic, or a hook.

## Start here

- [Quickstart](/docs/quickstart/) — key → run → result in under five minutes.
- [run()](/docs/primitives/run/) — the core primitive, field by field.
- [Runs & statistics](/docs/concepts/runs-and-statistics/) — why every result
  carries n and a CI, and how `auto(ci=0.95, moe=0.03)` sizes your run.
- [API reference](/docs/api-reference/) — generated from
  [/openapi.json](/openapi.json), examples on every operation.

## The primitives

| Primitive | What it answers |
| --- | --- |
| [`run()`](/docs/primitives/run/) | Does this policy work, on this robot, in this scene — with what precision? |
| [`eval()`](/docs/primitives/eval/) | How does it score on a frozen, citable benchmark suite? |
| [`verify()`](/docs/primitives/verify/) | Where does reality disagree with my simulator? |
| [`matrix()`](/docs/primitives/matrix/) | How does it hold up across embodiments × environments? |
| [`threshold()`](/docs/primitives/threshold/) | Iterate on cheap hardware until verified at a target — then prove it. |
| [`compare()`](/docs/primitives/compare/) | Is v4 actually better than v3, with paired initial conditions? |
| [`transfer()`](/docs/primitives/transfer/) | How much is lost moving to a new embodiment? |

## Built for agents

Every page here has a markdown twin at `<path>.md` and a "Copy page as
Markdown" button. [/llms.txt](/llms.txt) is the curated index;
[/llms-full.txt](/llms-full.txt) is the whole documentation as one file;
[/openapi.json](/openapi.json) is the machine-readable API. Code examples come
in four tabs — Python, TypeScript, cURL, and Agent (a tool-use JSON payload) —
generated from the same snippet source so they never drift. See the
[agent guide](/docs/sdks/agents/).

---

<!-- source: https://roborama.com/docs/quickstart/ -->

# Quickstart

Key → run → result in under five minutes.

You'll create an API key, launch a physical run on a humanoid in a kitchen
replica, and read back a result with a confidence interval. Nothing here
requires hardware knowledge — the defaults are the point.

### Install the SDK and set your key

Keys have the `rbr_live_` prefix and are read from the `ROBORAMA_API_KEY`
environment variable by every SDK and every example on this site.

```bash
pip install roborama-sdk    # or: npm i roborama
export ROBORAMA_API_KEY=rbr_live_...
```

### Launch a run

Pin a robot (`model@firmware`), an environment (`id@revision`), and a task.
`episodes="auto(ci=0.95, moe=0.03)"` sizes the run for a ±3-point margin of
error at 95% confidence — statistics as an input, not an afterthought. With no
`policy` attached, the run exercises the catalogue baseline policy for the
task, so you can see the whole loop before wiring up your own
([policy packaging](/docs/concepts/policies/) is step two).

*Example: your first run*

**Python**

```python
import roborama  # reads ROBORAMA_API_KEY from the environment

run = roborama.run(
    robot="g1-edu-pro@fw2.3",
    environment="kitchen-std@v1.2",
    task="pick_place@v1",
    episodes="auto(ci=0.95, moe=0.03)",
)
print(run.result())  # n=612  rate=0.874  ci95=(0.846, 0.898)
```

**TypeScript**

```typescript
import Roborama from "@roborama/sdk"; // reads ROBORAMA_API_KEY

const roborama = new Roborama();

const run = await roborama.runs.create({
  robot: "g1-edu-pro@fw2.3",
  environment: "kitchen-std@v1.2",
  task: "pick_place@v1",
  episodes: "auto(ci=0.95, moe=0.03)",
});

console.log(await run.result()); // n=612  rate=0.874  ci95=(0.846, 0.898)
```

**cURL**

```bash
curl https://api.roborama.com/v1/runs \
  -H "Authorization: Bearer $ROBORAMA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "kind": "run",
    "robot": "g1-edu-pro@fw2.3",
    "environment": "kitchen-std@v1.2",
    "task": "pick_place@v1",
    "episodes": "auto(ci=0.95, moe=0.03)"
  }'
```

**Agent (tool-use payload)**

```json
{
  "type": "tool_use",
  "name": "roborama_run",
  "input": {
    "robot": "g1-edu-pro@fw2.3",
    "environment": "kitchen-std@v1.2",
    "task": "pick_place@v1",
    "episodes": "auto(ci=0.95, moe=0.03)"
  }
}
```

### Read the result

The run queues, a cell schedules it, and episodes tick until the margin-of-error
target is met — here, n=612:

```text
n=612  rate=0.874  ci95=(0.846, 0.898)
```

A rate of 0.874 with a Wilson 95% interval of (0.846, 0.898): defensible in a
release review, reproducible from the recorded pins. Watch it live with
`run.watch()` or the [events stream](/docs/api-reference/runs/).

### Pull the artifacts

Every episode ships video, MCAP telemetry, and a re-runnable replay spec; the
run ships a citable verification report.

```python
import roborama

run = roborama.runs.get("run_8842")
print(run.episodes[17].video_url)
run.report(format="pdf")
```

## Where next

- The full engagement shape: [Advanced quickstart](/docs/quickstart-advanced/) —
  ship a kit, freeze your claim as a Task Spec, verify, and prove it across
  embodiments in seven calls.
- Attach your own policy: [Policies](/docs/concepts/policies/) — container,
  checkpoint, or endpoint, with contracts validated before any motor moves.
- Understand the numbers: [Runs & statistics](/docs/concepts/runs-and-statistics/).
- Price a bigger job first: [`POST /v1/quote`](/docs/api-reference/quote/) —
  both meters, visible in every quote.
- Gate your releases: [CI gates](/docs/guides/ci-gates/).

---

<!-- source: https://roborama.com/docs/quickstart-advanced/ -->

# Advanced quickstart

The end-to-end flow — package a checkpoint, ship a kit, freeze the claim as espresso@v1, verify the sim's doubts, and prove it across embodiments.

The [quickstart](/docs/quickstart/) proves the plumbing with a catalogue
task. This is the real shape of an engagement: a policy trained elsewhere, a
claim involving your own hardware, and evidence strong enough to release on.
Seven calls, start to finish.

## The whole flow

*Example: one claim, proven end to end*

**Python**

```python
import roborama  # reads ROBORAMA_API_KEY from the environment

# ── 1. Package the learnings ──────────────────────────────
policy = roborama.Policy.checkpoint(
    hf="pi/espresso-v7",                  # weights + config + norm stats
    runtime="openpi",
    inference={"action_horizon": 50, "chunk_size": 50, "temp": 0.0},
    action_space="joint_delta_50hz",
    observation_contract="droid-3cam",
)

# ── 2. Ship the hardware the claim depends on ─────────────
kit = roborama.kits.register(
    name="pi-breville", items=[{"desc": "Breville BES870", "qty": 2}])
kit.shipping_label()  # -> received -> tracked -> available

# ── 3. Formalize the claim as a Task Spec ──────────────────────────────
task = roborama.tasks.create(
    name="espresso", visibility="private",
    initial_conditions={
        "machine": {"object": "kit/pi-breville",
                    "pose": "P1", "tol_mm": 20},
        "cup": {"object": "catalogue/cup-std-08", "randomize": "zone-A"},
    },
    instructions={
        "sampled_per_episode": ["make an espresso", "brew me a coffee",
                                "fais un espresso"],
        "held_out": ["prepare a single shot"],
    },
    stages=[{"grasp_cup": "gt.pose(cup) in gripper"},
            {"cup_placed": "gt.pose(cup) within drip_zone"},
            {"extraction": "scale.delta > 0 within 60s"},
            {"served": "all success_predicates"}],
    success_predicates=[{"cup_on_tray": "gt.pose(cup) within tray_zone"},
                        {"liquid_mass": "scale.delta between 25 and 40 g"},
                        {"no_spill": "vision.spill_area < 2 cm2"},
                        {"t_complete": "episode.duration < 180 s"}],
    envelope={"lighting": ["3000K", "5600K"], "distractors": "set-B",
              "out_of_scope": ["oat_milk"]},
    baseline={"internal_trials": 30, "internal_rate": 0.87},
)

# ── 4. Agree on the method, then freeze it ──────────────────────────────
pilot = task.pilot(robot="g1-edu-pro@fw2.3", episodes=25)
print(pilot.stream_url)                    # watch the calibration live
task.freeze()  # -> espresso@v1, immutable, co-signed

# ── 5. Screen in your simulator; verify the flagged tail physically ─────
# the sim's uncertain slice, verified physically
scenarios = roborama.scenarios.from_file("sim_flagged_p1.json")
report = roborama.verify(
    policy=policy, task="espresso@v1",
    scenarios=scenarios,
    robots=["g1-edu-pro@fw2.3"],
    audit_sample=0.10, return_ground_truth=True,
)

# ── 6. Then prove the cross-embodiment claim itself ───────
job = roborama.matrix(
    policy=policy, task="espresso@v1",
    robots=["g1-edu-pro@fw2.3", "g1-edu-pro@fw2.4",
            "g1-edu-plus@fw2.3", "stretch3@fw1.9"],
    environments=["kitchen-std@v1.2", "kitchen-replica@v2.0"],
    episodes_per_cell="auto(ci=0.95, moe=0.05)",
    interventions="none",
)

# ── 7. The evidence ──────────────────────────────
r = job.result()
r.heatmap()                # embodiment × environment, rate ± CI per cell
r.stage_rates              # where in the task each embodiment degrades
r.instruction_breakdown    # held-out phrasings reported separately
r.verification_report(format="pdf")  # cites espresso@v1, fw pins,
                                     # out-of-scope verbatim
job.export(format="lerobot")  # episodes back into training
```

*(The TypeScript, cURL, and tool-use JSON variants of `quickstart-advanced` are on this page's own .md twin and in https://roborama.com/openapi.json.)*

## What each step buys you

1. **Package the learnings.** `Policy.checkpoint` pins more than weights:
   the `inference` dict (`action_horizon`, `chunk_size`, `temp`) is recorded
   with the run, because a decoding change can move a rate as much as a
   training change. Details in [Policies](/docs/concepts/policies/).
2. **Ship the hardware.** The espresso machine arrives as a
   [kit](/docs/concepts/tasks-and-method-transfer/): tracked (mocap markers,
   mass, mesh scan) and referenceable as `kit/pi-breville`.
3. **Formalize the claim.** The Task Spec turns "makes espresso" into
   instrument-bound predicates, a phrasing distribution with held-out
   paraphrases, staged progress scoring, and declared boundaries
   (`out_of_scope: ["oat_milk"]` appears verbatim on the report).
4. **Agree, then freeze.** The pilot is watched live; the freeze produces
   `espresso@v1` — immutable, mutually signed, cited by every report.
5. **Spend reality where the sim is unsure.** [verify()](/docs/primitives/verify/)
   runs the sim-flagged tail plus a random audit sample, and returns ground
   truth to recalibrate the simulator.
6. **Prove the claim itself.** A task-pinned [matrix()](/docs/primitives/matrix/)
   across four embodiments and two kitchens, `interventions="none"` declared
   up front so the statistics are untouched by helping hands.
7. **Collect the evidence.** Rate ± CI per cell, `stage_rates` showing where
   each embodiment degrades, `instruction_breakdown` with held-out phrasings
   separated, a citable verification report, and the episodes back in
   `lerobot` format for the next training round.

## Reading the espresso evidence

The canonical single-cell result, from
[/fixtures/runs.json](/fixtures/runs.json) — overall 0.850 (n=412,
ci95 0.812–0.881), zero interventions:

```text
stage_rates
  grasp_cup    0.966  n=412  ci95=(0.944, 0.980)
  cup_placed   0.932  n=412  ci95=(0.904, 0.953)
  extraction   0.893  n=412  ci95=(0.860, 0.919)
  served       0.850  n=412  ci95=(0.812, 0.881)

instruction_breakdown
  "make an espresso"       0.873  n=118  ci95=(0.801, 0.921)
  "brew me a coffee"       0.866  n=112  ci95=(0.791, 0.917)
  "fais un espresso"       0.848  n=99   ci95=(0.765, 0.906)
  "prepare a single shot"  0.795  n=83   ci95=(0.696, 0.868)   held-out
```

The stages localize the loss: grasping is nearly solved, extraction gives up
seven points, serving five more. The held-out phrasing runs about eight
points behind the sampled set — the gap between "understands the
instructions it trained on" and "understands the task."

> **Seven calls, one claim proven:** The learnings arrive as pinned weights, the claim leaves as a frozen
> protocol, and the evidence returns with error bars — on every body it names.

## Where next

- [Tasks & method transfer](/docs/concepts/tasks-and-method-transfer/) — the Task Spec in full.
- [CI gates](/docs/guides/ci-gates/) — make the matrix a release check.
- [threshold()](/docs/primitives/threshold/) — buy the outcome instead of the runs.

---

<!-- source: https://roborama.com/docs/concepts/runs-and-statistics/ -->

# Runs & statistics

Why every result carries n and a ci95, and how auto(ci=0.95, moe=0.03) sizes a run and stops it early.

A success rate is a claim about a binomial process: some number of episodes
ran, some succeeded. Strip away how many and how tight, and the number is
unfalsifiable. So every result carries its sample size and its interval —
`n` and `ci95` — a bare percentage is a bug, in the API and on this site.

## Reading a result

```text
n=612  success_rate=0.874  ci95=(0.846, 0.898)
failure_clusters: [grasp_slip: 41, perception_miss: 22, collision: 9]
robot_hours=20.4  environment_hours=20.4  cost_usd=3,812
artifacts: mcap[], video[], ground_truth[], report_pdf
```

Episode outcomes are binary, so success rates are binomial proportions. We
report Wilson score intervals because they behave correctly near 0 and 1 —
exactly where deployable policies live — where the textbook Wald interval
degenerates (an observed 1.0 gets a zero-width interval). Read
`ci95=(0.846, 0.898)` as: the observed rate is 0.874 (n=612), and rates
outside that interval are hard to square with the data at 95% confidence.

## Sizing a run with auto()

Pass an int to `episodes` if you already know your n.
`episodes="auto(ci=0.95, moe=0.03)"` solves for precision instead. It plans
conservatively from p=0.5, the worst case for binomial variance — n≈1,067
episodes for a ±3-point margin of error at 95% confidence — then sizes
sequentially: the Wilson interval is recomputed as episodes complete, and the
run stops as soon as the observed interval tightens under the target. The
canonical run stopped at n=612 with rate=0.874, ci95=(0.846, 0.898) — a
half-width of ≈0.026, under the 0.03 target and roughly 450 episodes short of
the plan. You pay for the precision you asked for, not for the worst case.

| Target moe | Planning n at p=0.5 |
| --- | --- |
| 0.05 (±5 pts) | ≈385 |
| 0.03 (±3 pts) | ≈1,068 |
| 0.02 (±2 pts) | ≈2,401 |

These are planning upper bounds — computed at the variance-maximizing p=0.5
and rounded up to whole episodes. Sequential stopping usually lands lower,
and the further the true rate sits from 0.5, the earlier the stop.

*(Example — auto-sized run: the `quickstart-run` code example appears in full earlier in this file, and on this page's own .md twin.)*

## Separating two policies

Precision on one rate is half the job; the other half is telling two policies
apart. Run them independently and the difference pays for both runs' noise.
[compare()](/docs/primitives/compare/) with `paired=True` runs both policies
against the same initial conditions, episode pair by episode pair, so the
shared variance cancels — pairing roughly halves the n needed to separate
them. In the worked example, v4 beats v3 by +4.1 points (p=0.008, 95% CI on
the effect +1.1 to +7.1 points).

## Failure clusters

The result doesn't stop at a rate. Failures arrive clustered — in the
canonical run, `grasp_slip: 41, perception_miss: 22, collision: 9`, which
accounts for 72 of its 77 failures. Clusters need not partition all failures:
the five that fit no cluster stay unlabeled rather than force-fitted. Every
cluster member links its episode artifacts — video, MCAP, replay spec — so
"what broke" is a queue of replayable episodes, not a guess. The artifacts
themselves are specified in the [data contract](/docs/concepts/data-contract/).

---

<!-- source: https://roborama.com/docs/concepts/robots-and-firmware/ -->

# Robots & firmware

Every embodiment is pinned as model@firmware, because a firmware bump is a different robot as far as your statistics are concerned.

A success rate is a property of a policy *on a specific machine running
specific firmware*. Change either and the number is a different claim. So
every robot in the API is addressed as `model@firmware` —
`g1-edu-pro@fw2.3` — and the pin travels with every result, every artifact,
and every citation.

## The fleet

*Example: roborama.robots.list()*

**Python**

```python
import roborama  # reads ROBORAMA_API_KEY from the environment

robots = roborama.robots.list()
print(robots)
# [{id:"g1-edu-pro", class:"humanoid", dof:37, hands:"dex3-1",
#   firmwares:["2.3","2.4"], cells:6, duty_cycle_pct:71, tier:"verify"}]

environments = roborama.environments.list(cls="kitchen")
print(environments)
# [{id:"kitchen-std", rev:"v1.2", instrumentation:["mocap","ft"],
#   objects:214, reset:"scripted", adder_tier:"replica"}]
```

**TypeScript**

```typescript
import Roborama from "@roborama/sdk"; // reads ROBORAMA_API_KEY

const roborama = new Roborama();

const robots = await roborama.robots.list();
console.log(robots);
// [{ id: "g1-edu-pro", class: "humanoid", dof: 37, hands: "dex3-1",
// firmwares: ["2.3", "2.4"], cells: 6,
//   duty_cycle_pct: 71, tier: "verify" }]

const environments = await roborama.environments.list({ cls: "kitchen" });
console.log(environments);
// [{ id: "kitchen-std", rev: "v1.2", instrumentation: ["mocap", "ft"],
//    objects: 214, reset: "scripted", adder_tier: "replica" }]
```

**cURL**

```bash
curl https://api.roborama.com/v1/robots \
  -H "Authorization: Bearer $ROBORAMA_API_KEY"

curl "https://api.roborama.com/v1/environments?cls=kitchen" \
  -H "Authorization: Bearer $ROBORAMA_API_KEY"
```

**Agent (tool-use payload)**

```json
[
  {
    "type": "tool_use",
    "name": "roborama_list_robots",
    "input": {}
  },
  {
    "type": "tool_use",
    "name": "roborama_list_environments",
    "input": { "cls": "kitchen" }
  }
]
```

Each entry describes capacity, not just capability: `cells` is how many
physical stations run that embodiment, and `duty_cycle_pct` is measured
utilization — how much of the wall-clock those cells spend running episodes
rather than resetting or idling. A `duty_cycle_pct` of 71 on six cells is
the honest answer to "how fast will my 600 episodes finish", and it feeds
the `queue_eta` in every [quote](/docs/api-reference/quote/).

| Field | Type | Notes |
| --- | --- | --- |
| `id` | `str` | model id, e.g. `g1-edu-pro`; pin firmware with `@`, e.g. `g1-edu-pro@fw2.3` |
| `class` | `str` | `humanoid`, `bimanual`, `mobile-manipulator`, `arm-pod` |
| `dof` | `int` | actuated degrees of freedom, hands included |
| `hands` | `str` | end effector fitted, e.g. `dex3-1`, `gripper-2f` |
| `firmwares` | `str[]` | pinnable firmware revisions currently installed on at least one cell |
| `cells` | `int` | physical stations running this embodiment |
| `duty_cycle_pct` | `int` | measured utilization; a throughput number, not a success rate |
| `tier` | `"verify" \| "soak"` | verification-tier hardware vs. commodity iteration pods |

## Why firmware is part of the address

Firmware changes controllers, and controllers change outcomes. The canonical
example is on the [matrix page](/docs/primitives/matrix/): the same policy on
`g1-edu-pro` in `kitchen-std@v1.2` scored 0.892 (n=240, ci95 0.846–0.925) on
`fw2.3` and 0.787 (n=240, ci95 0.731–0.835) on `fw2.4` — a 10.5-point drop
traced to a wrist controller change, surfacing as a `grasp_slip` cluster. If
the API let you say just "g1-edu-pro", those two results would be
indistinguishable, and one of them would be wrong.

`@latest` is accepted as a convenience, but the resolved pin is recorded in
the run and in every episode's `/meta` channel. There is no way to produce a
result that doesn't know what firmware it ran on.

> **Pins are recorded, not just requested:** `/meta` records the *realized* state per episode: resolved firmware,
> calibration age, actuator cycle counts, and thermal state. Two runs with the
> same pin are still distinguishable if the hardware drifted between them. See
> the [data contract](/docs/concepts/data-contract/).

## Tiers: verify and soak

The fleet splits into two tiers, and the split is the basis of the
[threshold contract](/docs/primitives/threshold/). Verification-tier hardware
(`g1-edu-pro`, `g1-edu-plus`, `spot-arm`, `aloha2-pro`, `tiago-pro`,
`aloha-bimanual`, `fr3-bench`, `stretch3`) is the machine your deployment
claim is about — instrumented, calibrated, and priced accordingly. Soak-tier
pods (`nori-a3` at 24 cells, `so-101` at 16, single-digit dollars per
robot-hour) exist for patient bulk iteration: cheap episodes while you climb,
escalating to verification tier when a threshold is crossed. Rates for both
are on [/pricing/](/pricing/).

## Where next

- [Environments](/docs/concepts/environments/) — the other half of the address.
- [run()](/docs/primitives/run/) — where the pin gets used.
- [API reference: Robots](/docs/api-reference/robots/) — the REST shape.

---

<!-- source: https://roborama.com/docs/concepts/environments/ -->

# Environments

Versioned catalogue scenes, from a bare bench cell to an instrumented surgical replica — with the reset problem handled for you.

An environment is the other half of a result's address. `kitchen-std@v1.2`
names a specific catalogue scene at a specific revision: the same 214
objects, the same layout envelope, the same instrumentation. When the scene
changes — an object swapped, a fixture moved — the revision changes, and
results across revisions stop being silently comparable.

## The catalogue

*(Example — roborama.environments.list(): the `discovery` code example appears in full earlier in this file, and on this page's own .md twin.)*

| Field | Type | Notes |
| --- | --- | --- |
| `id` | `str` | catalogue id; pin a revision with `@`, e.g. `kitchen-std@v1.2` |
| `rev` | `str` | scene revision; bumped whenever objects, layout envelope, or instrumentation change |
| `class` | `str` | `bench`, `kitchen`, `warehouse`, `surgical` — filter with `list(cls=...)` |
| `instrumentation` | `str[]` | `mocap` (6-DoF ground-truth poses), `ft` (wrist force-torque), `overhead-cam`, `high-speed-cam` |
| `objects` | `int` | catalogued objects in the scene's manifest |
| `reset` | `"scripted" \| "teleop-assisted"` | how the scene returns to initial conditions between episodes |
| `adder_tier` | `str` | pricing tier: `bare`, `standard`, `replica`, `instrumented-replica` |

The current catalogue spans the ladder: `cell-a` (a bare bench, 24 objects,
one overhead camera), `warehouse-std@v2.0` (388 objects under mocap),
`kitchen-std@v1.2` (a replica kitchen with mocap and force-torque sensing),
and `surgical-replica@v3` (61 objects, mocap, force-torque, and high-speed
cameras, with teleop-assisted resets).

## Instrumentation is what you're buying

The environment-hour meter prices what the scene can *measure*, which is why
`adder_tier` exists. A bare cell tells you whether the episode succeeded. An
instrumented replica also tells you *why*: mocap gives 6-DoF ground-truth
object poses on `/gt/object_poses`, force-torque gives contact evidence on
`/ft_wrist`. Those channels are what make failure clusters diagnosable and
what [verify()](/docs/primitives/verify/) returns to recalibrate a simulator.
Rates per tier are on [/pricing/](/pricing/).

## Resets are the throughput

Between every episode the scene has to return to initial conditions — objects
back on their marks, within the perturbation schedule's declared jitter. In
un-automated labs this is where the time goes: Physical Intelligence's
published online-RL result puts it at 7 of every 8 wall-clock minutes
(detailed on [/research/](/research/)). Catalogue scenes are built for scripted resets, so
episodes stream instead of trickle; `surgical-replica@v3` is the exception
where a human assists, and its `queue_eta` and adder tier reflect that.

> **Perturbation lives on the run, not the scene:** The revision pins what the scene *is*. How much each episode deviates —
> layout jitter, lighting, distractors — is the run's
> [perturbation schedule](/docs/primitives/run/), seeded and recorded per
> episode. Same scene revision + same schedule + same seed = the same
> distribution of initial conditions, replayable.

## Where next

- [Robots & firmware](/docs/concepts/robots-and-firmware/) — the embodiment half of the address.
- [quote](/docs/api-reference/quote/) — what a robot × environment × episodes job costs before you run it.
- [Data contract](/docs/concepts/data-contract/) — what the instrumentation records, channel by channel.

---

<!-- source: https://roborama.com/docs/concepts/policies/ -->

# Policies

Three ways to hand us a policy — container, checkpoint, endpoint — each behind the same declared contracts, validated before any motor moves.

A policy enters the facility in one of three packagings. All three sit behind
the same two declarations — an `action_space` and an `observation_contract` —
and both are validated against the robot and scene *before any motor moves*.
A policy that expects three cameras and gets two fails at submission with
`contract_validation_failed`, not at episode 40 with a mystery cluster.

## The three packagings

*Example: roborama.Policy*

**Python**

```python
import roborama  # reads ROBORAMA_API_KEY from the environment

# 1. Container — runs in-facility, GPU-adjacent
container = roborama.Policy.container(
    image="ghcr.io/acme/skill:v4",
    action_space="joint_delta_50hz",
    observation_contract="droid-3cam",
)

# 2. Checkpoint — known runtimes: openpi, lerobot
checkpoint = roborama.Policy.checkpoint(
    hf="acme/skill-v4", runtime="openpi")

# 3. Endpoint — customer-hosted inference; measured RTT logged per step
endpoint = roborama.Policy.endpoint(
    url="https://inference.acme.ai/act",
    latency_budget_ms=80,
    fallback="halt",
)

# Declared contracts are validated before any motor moves.
run = roborama.run(
    robot="g1-edu-pro@fw2.3",
    environment="kitchen-std@v1.2",
    policy=container,
    task="pick_place@v1",
    episodes="auto(ci=0.95, moe=0.03)",
)
print(run.id)
```

**TypeScript**

```typescript
import Roborama from "@roborama/sdk"; // reads ROBORAMA_API_KEY

const roborama = new Roborama();

// 1. Container — runs in-facility, GPU-adjacent
const container = {
  type: "container",
  image: "ghcr.io/acme/skill:v4",
  action_space: "joint_delta_50hz",
  observation_contract: "droid-3cam",
} as const;

// 2. Checkpoint — known runtimes: openpi, lerobot
const checkpoint = {
  type: "checkpoint",
  hf: "acme/skill-v4",
  runtime: "openpi",
} as const;

// 3. Endpoint — customer-hosted inference; measured RTT logged per step
const endpoint = {
  type: "endpoint",
  url: "https://inference.acme.ai/act",
  latency_budget_ms: 80,
  fallback: "halt",
} as const;

// Declared contracts are validated before any motor moves.
const run = await roborama.runs.create({
  robot: "g1-edu-pro@fw2.3",
  environment: "kitchen-std@v1.2",
  policy: container,
  task: "pick_place@v1",
  episodes: "auto(ci=0.95, moe=0.03)",
});
console.log(run.id, checkpoint.type, endpoint.type);
```

**cURL**

```bash
# Declared contracts are validated before any motor moves.
# policy.type: "container" | "checkpoint" | "endpoint"
curl https://api.roborama.com/v1/runs \
  -H "Authorization: Bearer $ROBORAMA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "kind": "run",
    "robot": "g1-edu-pro@fw2.3",
    "environment": "kitchen-std@v1.2",
    "policy": {
      "type": "endpoint",
      "url": "https://inference.acme.ai/act",
      "latency_budget_ms": 80,
      "fallback": "halt"
    },
    "task": "pick_place@v1",
    "episodes": "auto(ci=0.95, moe=0.03)"
  }'
```

**Agent (tool-use payload)**

```json
{
  "type": "tool_use",
  "name": "roborama_run",
  "input": {
    "robot": "g1-edu-pro@fw2.3",
    "environment": "kitchen-std@v1.2",
    "policy": {
      "type": "endpoint",
      "url": "https://inference.acme.ai/act",
      "latency_budget_ms": 80,
      "fallback": "halt"
    },
    "task": "pick_place@v1",
    "episodes": "auto(ci=0.95, moe=0.03)"
  }
}
```

| Packaging | Runs | Notes |
| --- | --- | --- |
| `Policy.container(image, action_space, observation_contract)` | in-facility, GPU-adjacent | any OCI image; lowest latency, no network in the loop |
| `Policy.checkpoint(hf=..., runtime=..., inference=...)` | in-facility, managed runtime | known runtimes: `openpi`, `lerobot`; the `inference` dict pins decoding |
| `Policy.endpoint(url, latency_budget_ms, fallback)` | your infrastructure | measured round-trip time logged per step; weights never leave your side |

**Container** is the default for anything custom: ship an OCI image, we run
it GPU-adjacent to the cell. **Checkpoint** is the low-friction path when
your policy already speaks a known runtime — point at a Hugging Face repo and
name the runtime. **Endpoint** inverts the trust: inference stays on your
infrastructure and the facility streams observations to your URL. That
choice is also an IP posture — see the
[data contract](/docs/concepts/data-contract/).

Checkpoints also pin how they're decoded. The `inference` dict —
`{"action_horizon": 50, "chunk_size": 50, "temp": 0.0}` — is recorded with
the run, because reproducibility includes decoding, not just weights: the
same checkpoint at a different action horizon is a different policy as far
as your statistics are concerned, the same way a firmware bump is a
different robot.

## Declared contracts

`action_space` names the command interface and rate the policy emits:
`joint_delta_50hz`, `joint_abs_20hz`, `ee_pose_10hz`, and the rest of the
catalogued spaces. `observation_contract` names the camera set and
proprioception schema it expects — `droid-3cam` is three RGB views plus the
joint-state schema. Validation checks both against the pinned robot and
scene: the declared space must be one the firmware accepts, and every feed
the contract names must exist in the cell.

The contracts are not paperwork; they are what makes a result attributable.
When a policy fails, you want to know it failed on physics, not on a silently
resampled action rate or a missing camera.

## Endpoint latency and the fallback

An endpoint policy puts your network in the control loop, so the latency
budget is explicit: `latency_budget_ms=80` means any step whose round trip
exceeds 80 ms triggers the declared `fallback` — `"halt"` freezes the arm and
ends the episode, which then lands in its own cluster rather than polluting
the physics statistics. Measured RTT is logged per step to `/events`, so you
can tell a policy problem from a network problem after the fact. Persistent
unreachability fails the run with `policy_endpoint_timeout`.

> **Budget the tail, not the mean:** A 40 ms median with a 200 ms p99 will halt episodes at a rate that dominates
> your failure statistics. Set `latency_budget_ms` from your p99, or package as
> a container and take the network out of the loop.

## Where next

- [run()](/docs/primitives/run/) — the call every packaging plugs into.
- [Data contract](/docs/concepts/data-contract/) — retention, `train_on_failures`, and who sees your weights.
- [Error codes](/docs/errors/) — `contract_validation_failed` and `policy_endpoint_timeout` in detail.

---

<!-- source: https://roborama.com/docs/concepts/tasks-and-method-transfer/ -->

# Tasks & method transfer

A claim becomes testable as a Task Spec — instrument-bound predicates, a piloted method, and a frozen, mutually signed revision.

"Our robot makes espresso" is not testable as stated. It becomes testable as
a **Task Spec**: a declarative, versioned protocol whose success predicates
are measurable by the cell's instrumentation. Onboarding a claim is a
first-class API flow, not a sales process — and the spec, not a conversation,
is what every verification report cites.

## Anatomy of a Task Spec

*Example: roborama.tasks.create() + lifecycle*

**Python**

```python
import roborama  # reads ROBORAMA_API_KEY from the environment

# A claim becomes testable as a Task Spec: declarative, versioned, with
# success predicates measurable by the cell's instrumentation.
task = roborama.tasks.create(
    name="espresso",
    visibility="private",  # "private" | "published" (comparable, citable)
    initial_conditions={
        "machine": {"object": "kit/acme-breville",
                    "pose": "P1", "tol_mm": 20},
        "cup": {"object": "catalogue/cup-std-08", "randomize": "zone-A"},
    },
    success_predicates=[               # each must bind to an instrument
        {"cup_on_tray": "gt.pose(cup) within tray_zone"},     # mocap
        {"liquid_mass": "scale.delta between 25 and 40 g"},   # load cell
        {"no_spill": "vision.spill_area < 2 cm2"},            # scene cams
        {"t_complete": "episode.duration < 180 s"},
    ],
    instructions={  # language-conditioned policies are tested
        "sampled_per_episode": [  # against a distribution, not one phrasing
            "make an espresso", "brew me a coffee", "fais un espresso",
        ],
        "held_out": ["prepare a single shot"],  # reported separately
    },
    stages=[                           # long-horizon tasks score progress
        {"grasp_cup": "gt.pose(cup) in gripper"},
        {"cup_placed": "gt.pose(cup) within drip_zone"},
        {"extraction": "scale.delta > 0 within 60s"},
        {"served": "all success_predicates"},
    ],
    envelope={                         # the claim's declared boundaries
        "lighting": ["3000K", "5600K"], "distractors": "set-B",
        "out_of_scope": ["oat_milk"],
    },
    baseline={"internal_trials": 30, "internal_rate": 0.87},  # private
)
print(task.status)  # "draft"

# Lifecycle — the protocol agreement, encoded:
pilot = task.pilot(robot="g1-edu-pro@fw2.3", episodes=25)  # calibration
print(pilot.stream_url)  # customer watches live

task.amend(  # iterate on the method while piloting
    success_predicates=[
        {"cup_on_tray": "gt.pose(cup) within tray_zone"},
        # widened after the pilot:
        {"liquid_mass": "scale.delta between 22 and 42 g"},
        {"no_spill": "vision.spill_area < 2 cm2"},
        {"t_complete": "episode.duration < 180 s"},
    ],
)

task.freeze()   # -> "espresso@v1": immutable, mutually signed;
                #    every verification report cites the frozen rev
print(task.status)  # "frozen", rev "espresso@v1"
```

*(The TypeScript, cURL, and tool-use JSON variants of `task-spec` are on this page's own .md twin and in https://roborama.com/openapi.json.)*

| Field | Notes |
| --- | --- |
| `visibility` | private tasks stay the customer's method; published tasks join the public catalogue, become cross-customer comparable, and amortize onboarding |
| `initial_conditions` | named objects with poses, tolerances, and randomization zones; kit hardware is referenced as `kit/<name>` |
| `success_predicates` | validated at creation: every predicate must bind to an available instrument in the target environment class |
| `instructions` | the phrasing distribution for language-conditioned policies; `held_out` paraphrases are never sampled during iteration and are reported separately |
| `stages` | ordered progress predicates; per-stage rates ship in results, so long-horizon tasks score progress, not just end-state |
| `envelope` | the claim's declared boundaries; out-of-scope declarations appear verbatim on reports |
| `baseline` | customer's internal trials, used to sanity-check the pilot and set expectations, never published |

Predicates bind to instruments, not to intentions: `cup_on_tray` reads mocap,
`liquid_mass` reads the load cell, `no_spill` reads the scene cameras. A
predicate that binds to no instrument in the target environment class fails
at creation with `contract_validation_failed` — the same before-any-motor-moves
discipline as [policy contracts](/docs/concepts/policies/).

## Lifecycle: draft → piloting → frozen

The lifecycle is a protocol agreement, encoded. `task.pilot()` runs a small
calibration session the customer watches live; `task.amend()` iterates on the
method while piloting — the worked example widens the `liquid_mass` band
after watching real pulls; `task.freeze()` produces `espresso@v1`: immutable
and mutually signed.

Frozen means frozen. Changes create `espresso@v2`, and results across
revisions are not silently comparable. A report against a frozen spec is
reproducible by construction and contestable only by contesting a protocol
the customer approved — the ISO 17025 method-transfer mechanism, as an API.

> **Pilot episodes are not evidence:** The pilot calibrates the method — the 25 episodes it runs never appear in a
> verification report. Evidence starts after the freeze, against the signed
> revision.

## Staged and language-conditioned results

A spec with `stages` gets `run.stage_rates` in results: per-stage rates,
each with its n and Wilson interval, so you see *where* a long-horizon task
degrades. A spec with `instructions` gets `run.instruction_breakdown`, with
held-out paraphrases reported separately. The canonical espresso run in
[/fixtures/runs.json](/fixtures/runs.json) shows both — held-out phrasing
0.795 (n=83, ci95 0.696–0.868) against the sampled set's best of 0.873
(n=118, ci95 0.801–0.921). The shapes are in the
[data contract](/docs/concepts/data-contract/).

## Object kits

When the claim involves the customer's own hardware, the hardware ships in
as a kit.

*Example: roborama.kits.register()*

**Python**

```python
import roborama  # reads ROBORAMA_API_KEY from the environment

# When the claim involves the customer's own hardware, ship it as a kit.
kit = roborama.kits.register(
    name="acme-breville",
    items=[{"desc": "Breville BES870", "qty": 1}],
)

print(kit.shipping_label())  # inbound label to the facility
print(kit.status)
# "received"  -> "tracked" (mocap markers, mass, mesh scan)
#             -> "available" as kit/acme-breville in environments

# Once available, reference it from a Task Spec's initial conditions:
#   {"machine": {"object": "kit/acme-breville", "pose": "P1", "tol_mm": 20}}
```

**TypeScript**

```typescript
import Roborama from "@roborama/sdk"; // reads ROBORAMA_API_KEY

const roborama = new Roborama();

// When the claim involves the customer's own hardware, ship it as a kit.
const kit = await roborama.kits.register({
  name: "acme-breville",
  items: [{ desc: "Breville BES870", qty: 1 }],
});

console.log(await kit.shippingLabel()); // inbound label to the facility
console.log(kit.status);
// "received"  -> "tracked" (mocap markers, mass, mesh scan)
//             -> "available" as kit/acme-breville in environments

// Once available, reference it from a Task Spec's initial conditions:
//   { machine: { object: "kit/acme-breville", pose: "P1", tol_mm: 20 } }
```

**cURL**

```bash
# Register the kit — the response includes the inbound shipping label URL
curl https://api.roborama.com/v1/kits \
  -H "Authorization: Bearer $ROBORAMA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "acme-breville",
    "items": [{"desc": "Breville BES870", "qty": 1}]
  }'

# Poll the status progression: received -> tracked -> available
curl https://api.roborama.com/v1/kits/kit_acme_breville \
  -H "Authorization: Bearer $ROBORAMA_API_KEY"
```

**Agent (tool-use payload)**

```json
{
  "type": "tool_use",
  "name": "roborama_kits_register",
  "input": {
    "name": "acme-breville",
    "items": [{ "desc": "Breville BES870", "qty": 1 }]
  }
}
```

Status progresses `received → tracked → available`: on arrival the machine
gets mocap markers, a mass measurement, and a mesh scan, then becomes
referenceable from Task Specs as `kit/acme-breville`. Kits integrate into
the standard fixture/tracking/reset system — the system is never customized
around a kit.

## Where next

- [Advanced quickstart](/docs/quickstart-advanced/) — the full flow: kit → spec → freeze → verify → matrix.
- [run()](/docs/primitives/run/) — where the frozen `task` pin gets used.
- [API reference: Tasks](/docs/api-reference/tasks/) — create, pilot, amend, freeze on the wire.

---

<!-- source: https://roborama.com/docs/concepts/data-contract/ -->

# Data contract

What every episode records, channel by channel — plus retention, export formats, the IP boundary, and receipted deletion.

Every episode produces the same record: an MCAP file with a fixed channel
set, a video, and a replay spec. The contract is deliberately boring — the
value of episode 17 is that it's shaped exactly like episode 16 — and it's
the substrate everything else stands on: failure clusters link into it,
[verify()](/docs/primitives/verify/) returns it as ground truth, and the
verification report cites it.

## The episode record

*Example: episodes, exports, reports*

**Python**

```python
import roborama  # reads ROBORAMA_API_KEY from the environment

run = roborama.runs.get("run_8842")

episode = run.episodes[17]
print(episode.video_url)
print(episode.mcap_url)     # channels: /joint_states_measured,
                            # /joint_targets_commanded, /camera/*,
                            # /ft_wrist, /gt/object_poses, /events, /meta
print(episode.replay_spec)  # initial conditions, re-runnable

run.export(format="lerobot")   # also: "rlds", "mcap-bundle"
run.report(format="pdf")       # the deployment verification report
```

**TypeScript**

```typescript
import Roborama from "@roborama/sdk"; // reads ROBORAMA_API_KEY

const roborama = new Roborama();

const run = await roborama.runs.get("run_8842");

const episodes = await run.episodes();
const episode = episodes[17];
console.log(episode.video_url);
// channels: /joint_states_measured, /joint_targets_commanded,
console.log(episode.mcap_url);
//   /camera/*, /ft_wrist, /gt/object_poses, /events, /meta
console.log(episode.replay_spec); // initial conditions, re-runnable

await run.export({ format: "lerobot" }); // also: "rlds", "mcap-bundle"
await run.report({ format: "pdf" }); // the deployment verification report
```

**cURL**

```bash
curl https://api.roborama.com/v1/runs/run_8842/episodes \
  -H "Authorization: Bearer $ROBORAMA_API_KEY"

curl "https://api.roborama.com/v1/runs/run_8842/export?format=lerobot" \
  -X POST \
  -H "Authorization: Bearer $ROBORAMA_API_KEY"

curl "https://api.roborama.com/v1/runs/run_8842/report?format=pdf" \
  -H "Authorization: Bearer $ROBORAMA_API_KEY" \
  -o verification-report.pdf
```

**Agent (tool-use payload)**

```json
[
  {
    "type": "tool_use",
    "name": "roborama_get_episodes",
    "input": { "run_id": "run_8842" }
  },
  {
    "type": "tool_use",
    "name": "roborama_export_run",
    "input": { "run_id": "run_8842", "format": "lerobot" }
  },
  {
    "type": "tool_use",
    "name": "roborama_get_report",
    "input": { "run_id": "run_8842", "format": "pdf" }
  }
]
```

| MCAP channel | Contents |
| --- | --- |
| `/joint_states_measured` | encoder-measured joint positions, velocities, efforts |
| `/joint_targets_commanded` | what the policy asked for — diff against measured to see tracking error |
| `/camera/*` | every feed named by the observation contract, timestamped |
| `/ft_wrist` | wrist force-torque, where the environment is instrumented for it |
| `/gt/object_poses` | mocap 6-DoF ground-truth object poses — the channel sim calibration wants |
| `/events` | episode lifecycle: start, success/failure determination, cluster label, endpoint RTTs, estops |
| `/meta` | reproducibility record — see below |

`/meta` is the channel that makes a result defensible later: resolved
firmware pin, calibration age, actuator cycle counts, thermal state, the
seed, and the *realized* perturbations — the layout jitter that actually
happened, not just what the schedule requested.

Each episode also carries `replay_spec`: its initial conditions in a form you
can re-run on the same scene revision or load into a simulator.

## Staged and instruction-level statistics

Runs against a [Task Spec](/docs/concepts/tasks-and-method-transfer/) that
declares stages or an instruction distribution carry two extra result
accessors, both under the same statistics discipline as the headline rate:

- `run.stage_rates` — per-stage success, each stage with its own n and
  Wilson ci95, so a long-horizon task shows *where* it degrades.
- `run.instruction_breakdown` — rate per instruction phrasing, held-out
  paraphrases reported separately, each row with n and ci95.

The result also records the realized `interventions` block (declared
policy, count, timestamps); intervened episodes are flagged in the
statistics. The canonical espresso run in
[/fixtures/runs.json](/fixtures/runs.json) carries all three.

## Exports

`run.export(format=...)` repackages the episode set: `"lerobot"` and
`"rlds"` for training pipelines, `"mcap-bundle"` for the raw record.
`run.report(format="pdf")` renders the deployment verification report — in
our terminology this is *verification* producing *conformance evidence*, and
the report says so.

## Retention, ownership, deletion

`data.retention` on the run sets how long artifacts live (the canonical run
uses `"30d"`). Fetching after expiry returns `retention_expired`.
`roborama.data.purge(run_id)` deletes early — customer-initiated, and it
returns a receipt with an id you can point an auditor at.

`data.train_on_failures` defaults to `False`, contractually: your episodes —
failures included — are your IP, not our training set, unless you flip the
bit. Endpoint-packaged [policies](/docs/concepts/policies/) extend the same
posture to weights, which never enter the facility at all.

> **The default is the contract:** The `False` default on `train_on_failures` is a term of service, not a
> configuration suggestion. No run trains on your failures because you forgot
> a flag.

## Calibration exports

There is one deliberately scoped exception to "your episodes are yours
alone": aggregate calibration data. Simulator vendors can license
ground-truth channels — `gt/object_poses`, commanded-vs-executed,
contact events — under `sim-calibration-v1`, which permits validation use
and forbids policy training.

*Example: roborama.calibration.export()*

**Python**

```python
import roborama  # reads ROBORAMA_API_KEY from the environment

ds = roborama.calibration.export(
    embodiment="g1-edu-pro",
    scenes=["kitchen-std@v1.2"],
    channels=["gt/object_poses", "commanded_vs_executed", "contact_events"],
    license="sim-calibration-v1",  # scoped: validation only, never training
)

print(ds.id, ds.download_url)
```

**TypeScript**

```typescript
import Roborama from "@roborama/sdk"; // reads ROBORAMA_API_KEY

const roborama = new Roborama();

const ds = await roborama.calibration.export({
  embodiment: "g1-edu-pro",
  scenes: ["kitchen-std@v1.2"],
  channels: ["gt/object_poses", "commanded_vs_executed", "contact_events"],
  license: "sim-calibration-v1", // scoped: validation only, never training
});

console.log(ds.id, ds.download_url);
```

**cURL**

```bash
curl https://api.roborama.com/v1/calibration/exports \
  -H "Authorization: Bearer $ROBORAMA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "embodiment": "g1-edu-pro",
    "scenes": ["kitchen-std@v1.2"],
    "channels": ["gt/object_poses", "commanded_vs_executed",
                 "contact_events"],
    "license": "sim-calibration-v1"
  }'
```

**Agent (tool-use payload)**

```json
{
  "type": "tool_use",
  "name": "roborama_calibration_export",
  "input": {
    "embodiment": "g1-edu-pro",
    "scenes": ["kitchen-std@v1.2"],
    "channels": ["gt/object_poses", "commanded_vs_executed",
                 "contact_events"],
    "license": "sim-calibration-v1"
  }
}
```

## Where next

- [Sim triage](/docs/guides/sim-triage/) — feeding disagreement ground truth back into a simulator.
- [run()](/docs/primitives/run/) — where `data.retention` and `train_on_failures` are set.
- [Billing & budgets](/docs/guides/billing-and-budgets/) — purge receipts, key scopes, governance.

---

<!-- source: https://roborama.com/docs/primitives/run/ -->

# run()

The core primitive — one policy, one robot, one scene, and back comes a rate with n, a Wilson interval, and named failure clusters.

`roborama.run()` puts a policy on a physical robot in a versioned scene and
returns a statistic you can defend: a success rate with its sample size and
Wilson 95% interval, failure clusters, both meters, and the full episode
record. Every other primitive is a composition of this call. Everything else
on this page is a field on it.

## The call

*Example: roborama.run()*

**Python**

```python
import roborama  # reads ROBORAMA_API_KEY from the environment

run = roborama.run(
    robot="g1-edu-pro@fw2.3",            # embodiment @ pinned firmware
    environment="kitchen-std@v1.2",      # versioned catalogue scene
    policy=roborama.Policy.container(    # see policy packaging
        image="ghcr.io/acme/skill:v4",
        action_space="joint_delta_50hz",
        observation_contract="droid-3cam",
    ),
    task="load_dishwasher@v2",
    episodes="auto(ci=0.95, moe=0.03)",  # size n for ±3% at 95% — or an int
    perturbation={                       # the schedule IS the product spec
        "layout_jitter_mm": 25,
        "lighting": ["3000K", "5600K"],
        "distractors": "set-B",
        "seed": 42,
    },
    data={"retention": "30d", "train_on_failures": False},  # IP posture
    max_budget_usd=4_000,                # hard stop, metered live
)

print(run.result())
#  n=612  success_rate=0.874  ci95=(0.846, 0.898)
#  failure_clusters: [grasp_slip: 41, perception_miss: 22, collision: 9]
#  robot_hours=20.4  environment_hours=20.4  cost_usd=3_812
#  artifacts: mcap[], video[], ground_truth[], report_pdf
```

**TypeScript**

```typescript
import Roborama from "@roborama/sdk"; // reads ROBORAMA_API_KEY

const roborama = new Roborama();

const run = await roborama.runs.create({
  robot: "g1-edu-pro@fw2.3", // embodiment @ pinned firmware
  environment: "kitchen-std@v1.2", // versioned catalogue scene
  policy: {
    type: "container",
    image: "ghcr.io/acme/skill:v4",
    action_space: "joint_delta_50hz",
    observation_contract: "droid-3cam",
  },
  task: "load_dishwasher@v2",
  episodes: "auto(ci=0.95, moe=0.03)", // size n for ±3% at 95%
  perturbation: {
    layout_jitter_mm: 25,
    lighting: ["3000K", "5600K"],
    distractors: "set-B",
    seed: 42,
  },
  data: { retention: "30d", train_on_failures: false },
  max_budget_usd: 4000, // hard stop, metered live
});

const result = await run.result();
console.log(result);
// { n: 612, success_rate: 0.874, ci95: [0.846, 0.898], cost_usd: 3812 }
```

**cURL**

```bash
curl https://api.roborama.com/v1/runs \
  -H "Authorization: Bearer $ROBORAMA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "kind": "run",
    "robot": "g1-edu-pro@fw2.3",
    "environment": "kitchen-std@v1.2",
    "policy": {
      "type": "container",
      "image": "ghcr.io/acme/skill:v4",
      "action_space": "joint_delta_50hz",
      "observation_contract": "droid-3cam"
    },
    "task": "load_dishwasher@v2",
    "episodes": "auto(ci=0.95, moe=0.03)",
    "perturbation": {
      "layout_jitter_mm": 25,
      "lighting": ["3000K", "5600K"],
      "distractors": "set-B",
      "seed": 42
    },
    "data": { "retention": "30d", "train_on_failures": false },
    "max_budget_usd": 4000
  }'
```

**Agent (tool-use payload)**

```json
{
  "type": "tool_use",
  "name": "roborama_run",
  "input": {
    "robot": "g1-edu-pro@fw2.3",
    "environment": "kitchen-std@v1.2",
    "policy": {
      "type": "container",
      "image": "ghcr.io/acme/skill:v4",
      "action_space": "joint_delta_50hz",
      "observation_contract": "droid-3cam"
    },
    "task": "load_dishwasher@v2",
    "episodes": "auto(ci=0.95, moe=0.03)",
    "perturbation": {
      "layout_jitter_mm": 25,
      "lighting": ["3000K", "5600K"],
      "distractors": "set-B",
      "seed": 42
    },
    "data": { "retention": "30d", "train_on_failures": false },
    "max_budget_usd": 4000
  }
}
```

```text
n=612  success_rate=0.874  ci95=(0.846, 0.898)
failure_clusters: [grasp_slip: 41, perception_miss: 22, collision: 9]
robot_hours=20.4  environment_hours=20.4  cost_usd=3,812
artifacts: mcap[], video[], ground_truth[], report_pdf
```

Every field of that result is load-bearing: the rate ships as 0.874 with n=612
and ci95=(0.846, 0.898), the failures arrive pre-clustered
(grasp_slip: 41, perception_miss: 22, collision: 9), and both meters are on
the receipt.

## Fields

| Field | Type | Notes |
| --- | --- | --- |
| `robot` | `str` | `model@firmware`; `@latest` is allowed but the resolved pin is recorded |
| `environment` | `str` | catalogue id + revision, from bare `cell-a` to `surgical-replica@v3` |
| `policy` | `Policy` | `container`, `checkpoint`, or `endpoint`; contracts validated before any motor moves |
| `task` | `str` | versioned task id, e.g. `load_dishwasher@v2` |
| `episodes` | `int \| "auto(...)"` | auto-sizing solves for margin of error — statistics as an input, not an afterthought |
| `perturbation` | `dict` | seeded, versioned, replayable; declared in the result |
| `data.retention` | `str` | artifact retention window, e.g. `"30d"`; early deletion via `data.purge` is receipted |
| `data.train_on_failures` | `bool` | customer IP boundary; contractual default `False` |
| `max_budget_usd` | `int` | hard stop, metered live against both meters |
| `priority` | `"burst" \| "standard" \| "soak"` | maps to the rate-card tiers |
| `interventions` | `"none" \| "on_stall" \| "scripted"` | declared before the run; counts and timestamps ship in results, and any intervened episode is flagged in the statistics |

## Auto-sizing episodes

Pass an int if you already know your n. `episodes="auto(ci=0.95, moe=0.03)"`
solves for it instead: episodes keep scheduling until the margin of error
tightens to ±3 points at 95% confidence. The canonical run above stopped at
n=612. How the sizing works, and when a tighter margin is worth the episodes,
is covered in [Runs & statistics](/docs/concepts/runs-and-statistics/).

## The perturbation schedule

The schedule is the product spec. `layout_jitter_mm: 25` is a claim about the
messiness of the kitchens your policy is supposed to survive; the lighting
list and distractor set are the rest of that claim. The schedule is seeded and
versioned, so the same schedule and seed reproduce the same distribution of
scenes — which is what makes two runs comparable and a result replayable. The
realized values, not just the requested ones, are recorded per episode in the
`/meta` channel, alongside firmware, calibration age, and thermal state.

## Data posture

`data.train_on_failures` defaults to `False`, and the default is contractual:
your failure episodes are your IP, not our training set, unless you flip the
bit yourself. `data.retention` sets how long artifacts live;
`roborama.data.purge(run_id)` deletes early and returns a receipt. The full
posture — what comes back, who owns it, what we may touch — is specified in
the [data contract](/docs/concepts/data-contract/).

## Budget and priority

`max_budget_usd` is a hard stop, metered live against robot-hours and
environment-hours. A run that hits it halts with `budget_exceeded` and keeps
the partial result: n and the interval simply describe the episodes that ran.
The canonical run was capped at 4,000 and finished at 3,812.

> **A capped run is not a lost run:** `budget_exceeded` is a status, not a failure. You keep every episode,
> artifact, and statistic accrued up to the stop — often enough to decide
> whether the remaining margin is worth buying.

`priority` picks the queue: `"burst"` when you need the answer today,
`"standard"` for the default rate, `"soak"` for patient bulk iteration on
commodity cells.

## Interventions

Human help is a statistic-shaped hole unless it's declared.
`interventions` sets the policy before the run: `"none"` (the default —
a stalled episode is a failed episode), `"on_stall"` (an operator may
recover a stall), or `"scripted"` (declared recovery procedures only).
Whatever you declare, the counts and timestamps ship in the result, and any
intervened episode is flagged in the statistics — a rate propped up by
helping hands can't masquerade as autonomy. The canonical espresso run in
[/fixtures/runs.json](/fixtures/runs.json) declares `"none"` and reports
`count: 0`.

For tasks whose [Task Spec](/docs/concepts/tasks-and-method-transfer/)
declares stages or an instruction distribution, the result also carries
`stage_rates` and `instruction_breakdown` — each entry with its own n and
ci95.

## Where next

- [eval()](/docs/primitives/eval/) — the same discipline on a frozen, citable suite.
- [matrix()](/docs/primitives/matrix/) — this call, swept across embodiments × environments.
- [API reference](/docs/api-reference/) — the REST shape of `run()`, generated from [/openapi.json](/openapi.json).

---

<!-- source: https://roborama.com/docs/primitives/eval/ -->

# eval()

Frozen benchmark suites with attested, citable reports — comparable across every team that ran the same suite revision.

`roborama.run()` answers a question about your task. `roborama.eval()` answers
a question other people can check. It runs your policy against a frozen
benchmark suite and returns a report with an attestation anyone can verify —
no Roborama account required.

## The call

*Example: roborama.eval()*

**Python**

```python
import roborama  # reads ROBORAMA_API_KEY from the environment

policy = roborama.Policy.checkpoint(hf="acme/skill-v4", runtime="openpi")

report = roborama.eval(
    policy=policy,
    suite="rbr-manip-core@v3",  # versioned suite (5 tasks × 300 eps)
    robots=["g1-edu-pro@fw2.3"],
    publish="private",           # "private" | "leaderboard" (opt-in only)
)

print(report.citation)
# Roborama Manip-Core v3, run 2026-09-04, attestation rbr:att:9f3c2ab7
print(report.attestation_url)   # verifiable benchmark attestation
```

**TypeScript**

```typescript
import Roborama from "@roborama/sdk"; // reads ROBORAMA_API_KEY

const roborama = new Roborama();

const report = await roborama.runs.create({
  kind: "eval",
  policy: { type: "checkpoint", hf: "acme/skill-v4", runtime: "openpi" },
  suite: "rbr-manip-core@v3", // versioned suite (5 tasks × 300 eps)
  robots: ["g1-edu-pro@fw2.3"],
  publish: "private", // "private" | "leaderboard" (opt-in only)
});

console.log(report.citation);
// Roborama Manip-Core v3, run 2026-09-04, attestation rbr:att:9f3c2ab7
console.log(report.attestation_url);
```

**cURL**

```bash
curl https://api.roborama.com/v1/runs \
  -H "Authorization: Bearer $ROBORAMA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "kind": "eval",
    "policy": { "type": "checkpoint", "hf": "acme/skill-v4",
                "runtime": "openpi" },
    "suite": "rbr-manip-core@v3",
    "robots": ["g1-edu-pro@fw2.3"],
    "publish": "private"
  }'
```

**Agent (tool-use payload)**

```json
{
  "type": "tool_use",
  "name": "roborama_eval",
  "input": {
    "policy": { "type": "checkpoint", "hf": "acme/skill-v4",
                "runtime": "openpi" },
    "suite": "rbr-manip-core@v3",
    "robots": ["g1-edu-pro@fw2.3"],
    "publish": "private"
  }
}
```

## Frozen suites

A suite is a frozen bundle: tasks, environments, perturbation schedules, and
scoring, pinned under one revision. `rbr-manip-core@v3` is five tasks × 300
episodes — `pick_place@v1`, `load_dishwasher@v2`, `shelf_restock@v1`,
`fold_towel@v2`, `bin_pick@v1` — across `kitchen-std@v1.2`,
`warehouse-std@v2.0`, and `cell-a@v1.0`, with perturbation schedules pinned at
`suite-core@v3.0` and binary success scored with Wilson 95% intervals. The
bundle froze on 2026-03-14 and nothing in it moves: a v3 score next year means
what a v3 score means today. When the suite needs to change, that is a new
revision, and revisions are never compared to each other.

> **Comparable by construction:** Two teams that run the same suite revision ran the same tasks, the same
> scenes, the same perturbation schedules, and the same scoring. Their numbers
> sit on one axis. That is the property that makes independent citation
> possible.

## A sample report

The canonical `g1-edu-pro@fw2.3` report on `rbr-manip-core@v3`:

| Task | n | Success rate | ci95 |
| --- | --- | --- | --- |
| `pick_place@v1` | 300 | 0.903 | 0.865–0.932 |
| `load_dishwasher@v2` | 300 | 0.873 | 0.831–0.906 |
| `shelf_restock@v1` | 300 | 0.813 | 0.765–0.853 |
| `fold_towel@v2` | 300 | 0.93 | 0.895–0.954 |
| `bin_pick@v1` | 300 | 0.777 | 0.726–0.82 |
| Overall | 1500 | 0.859 | 0.841–0.876 |

The per-task rows are the point. An overall 0.859 (n=1500, ci95 0.841–0.876)
looks healthy; the table shows `bin_pick@v1` at 0.777 (n=300, ci95 0.726–0.82)
dragging it, which is where your next training run should look.

## Citation and attestation

`report.citation` returns a string built for a paper or a launch post:

```text
Roborama Manip-Core v3, run 2026-09-04, attestation rbr:att:9f3c2ab7
```

The attestation id `rbr:att:9f3c2ab7` resolves at `report.attestation_url` —
`https://api.roborama.com/v1/attestations/rbr:att:9f3c2ab7` — where anyone can
check the n, the rates, the intervals, and every pin behind them. A claim that
carries its own audit trail.

## Publishing

Results are private. `publish="private"` is the default posture, and
`publish="leaderboard"` is strictly opt-in: it places the attested report on
the public leaderboard for that suite revision, where it is comparable with
every other opted-in result on the same revision. Nothing is published by
accident; a key without the right scope gets `publish_forbidden` back.

## Where next

- [run()](/docs/primitives/run/) — your own task, your own perturbation schedule.
- [matrix()](/docs/primitives/matrix/) — coverage across embodiments × environments.
- [CI gates](/docs/guides/ci-gates/) — run the suite on `release/*` and fail the build on a drop.

---

<!-- source: https://roborama.com/docs/primitives/verify/ -->

# verify()

Run the scenarios your simulator flagged on instrumented hardware — get back agreement, disagreements, and ground truth to recalibrate.

Simulation screens; physical testing verifies. `roborama.verify()` is the
interlock between the two: it consumes the initial conditions your simulator
is least sure about, runs them on instrumented hardware, and returns where
reality agreed with the sim — and, more usefully, where it didn't, with the
ground truth to fix it.

## The call

*Example: roborama.verify()*

**Python**

```python
import roborama  # reads ROBORAMA_API_KEY from the environment

policy = roborama.Policy.checkpoint(hf="acme/skill-v4", runtime="openpi")

# Initial conditions your simulator marked uncertain: sampled states from
# PolaRiS / Isaac / world-model exports, or Roborama layout-replay specs.
scenarios = roborama.scenarios.from_file("sim_flagged_p1.json")

report = roborama.verify(
    policy=policy,
    scenarios=scenarios,
    robots=["g1-edu-pro@fw2.3", "g1-edu-plus@fw2.3"],
    audit_sample=0.10,          # random episodes vs. the sim's blind spots
    return_ground_truth=True,   # mocap 6-DoF poses + commanded-vs-executed,
                                # formatted to recalibrate your simulator
)

print(report.sim_agreement)  # where reality matched sim predictions
print(report.disagreements)  # scenario ids where reality diverged
```

**TypeScript**

```typescript
import { readFileSync } from "node:fs";
import Roborama from "@roborama/sdk"; // reads ROBORAMA_API_KEY

const roborama = new Roborama();

// Initial conditions your simulator marked uncertain: sampled states from
// PolaRiS / Isaac / world-model exports, or Roborama layout-replay specs.
const scenarios = JSON.parse(readFileSync("sim_flagged_p1.json", "utf8"));

const report = await roborama.runs.create({
  kind: "verify",
  policy: { type: "checkpoint", hf: "acme/skill-v4", runtime: "openpi" },
  scenarios,
  robots: ["g1-edu-pro@fw2.3", "g1-edu-plus@fw2.3"],
  audit_sample: 0.1, // random episodes vs. the sim's blind spots
  return_ground_truth: true, // mocap 6-DoF poses + commanded-vs-executed
});

console.log(report.sim_agreement); // where reality matched sim predictions
console.log(report.disagreements); // scenario ids where reality diverged
```

**cURL**

```bash
# sim_flagged_p1.json: initial conditions your simulator marked uncertain
# (PolaRiS / Isaac / world-model exports, or Roborama layout-replay specs)
curl https://api.roborama.com/v1/runs \
  -H "Authorization: Bearer $ROBORAMA_API_KEY" \
  -H "Content-Type: application/json" \
  -d "$(jq -n --slurpfile s sim_flagged_p1.json '{
    kind: "verify",
    policy: { type: "checkpoint", hf: "acme/skill-v4", runtime: "openpi" },
    scenarios: $s[0],
    robots: ["g1-edu-pro@fw2.3", "g1-edu-plus@fw2.3"],
    audit_sample: 0.10,
    return_ground_truth: true
  }')"
```

**Agent (tool-use payload)**

```json
{
  "type": "tool_use",
  "name": "roborama_verify",
  "input": {
    "policy": { "type": "checkpoint", "hf": "acme/skill-v4",
                "runtime": "openpi" },
    "scenarios": {
      "format": "layout-replay",
      "source": "sim_flagged_p1.json",
      "count": 318
    },
    "robots": ["g1-edu-pro@fw2.3", "g1-edu-plus@fw2.3"],
    "audit_sample": 0.1,
    "return_ground_truth": true
  }
}
```

## Scenarios your sim flagged

`scenarios` accepts sampled initial states in PolaRiS, Isaac, and world-model
formats, or Roborama layout-replay specs — scene layouts as JSON or USD. The
usual source is something like `sim_report.flagged(percentile=1)`: the slice
of scenarios where your simulator's own confidence is worst. That targeting is
the economics of the primitive — robot-hours go to the episodes where the sim
is least trustworthy instead of being spread uniformly over scenes the sim
already handles well. A malformed scene file fails fast with
`scenario_format_invalid`, before anything is scheduled.

## Auditing the blind spots

A flagged set has a failure mode of its own: it only contains what the
simulator knows it doesn't know. `audit_sample=0.10` spends a tenth of the
episode budget on randomly drawn scenarios instead — a control arm against the
sim's blind spots. If the audit arm disagrees with the sim more than the
flagged arm does, your simulator's uncertainty estimates are themselves
miscalibrated, and that is worth knowing before you trust its next screen.

## Ground truth that recalibrates

With `return_ground_truth=True`, every episode ships motion-capture 6-DoF
object poses and commanded-versus-executed joint traces — in MCAP channels
`/gt/object_poses`, `/joint_targets_commanded`, and `/joint_states_measured` —
formatted to load into a simulator calibration pipeline. You don't just learn
that reality diverged; you get the trajectories to make your simulator
diverge less next time.

## Agreement, and the payload

`report.sim_agreement` is the headline: 0.918 (n=318, ci95 0.883–0.944) of
physical outcomes matched the sim's predictions on this set.
`report.disagreements` is the payload: the scenario ids where reality
diverged, each shipping with full ground truth. Read them together — agreement
tells you how far to trust the simulator, disagreements tell you exactly
which scenarios to re-examine and what actually happened in them. This is the
calibration loop that makes sim vendors customers.

## Where next

- [Sim triage](/docs/guides/sim-triage/) — the workflow: flag, verify, recalibrate, repeat.
- [Data contract](/docs/concepts/data-contract/) — what comes back, in which formats, and who owns it.
- [run()](/docs/primitives/run/) — the underlying primitive, field by field.

---

<!-- source: https://roborama.com/docs/primitives/matrix/ -->

# matrix()

One policy swept across embodiments × environments — rate and interval per cell, regression diffs, and a verification report.

A policy that works is a claim about one robot in one scene. A release
decision needs the grid. `roborama.matrix()` sweeps the same task across every
embodiment × environment pair you name and returns a rate with its interval
per cell — plus the diffs and the document that release engineering actually
asked for.

## The call

*Example: roborama.matrix()*

**Python**

```python
import roborama  # reads ROBORAMA_API_KEY from the environment

policy = roborama.Policy.checkpoint(hf="acme/skill-v4", runtime="openpi")

job = roborama.matrix(
    policy=policy,
    robots=["g1-edu-pro@fw2.3", "g1-edu-pro@fw2.4", "g1-edu-plus@fw2.3",
            "aloha2-pro@fw1.1", "spot-arm@fw4.1", "tiago-pro@fw2.0",
            "fr3-bench@fw5.2", "stretch3@fw1.9", "nori-a3@fw1.0",
            "so-101@fw1.2"],
    environments=["cell-a", "kitchen-std@v1.2", "warehouse-std@v2.0"],
    task="pick_place_class3",
    episodes_per_cell="auto(ci=0.95, moe=0.05)",
)

job.heatmap()                    # embodiment × environment grid of rate±CI
job.regressions(vs="run_8821")   # cells that degraded vs. a prior release
job.verification_report(format="pdf")  # the release-gate deliverable
```

**TypeScript**

```typescript
import Roborama from "@roborama/sdk"; // reads ROBORAMA_API_KEY

const roborama = new Roborama();

const job = await roborama.runs.create({
  kind: "matrix",
  policy: { type: "checkpoint", hf: "acme/skill-v4", runtime: "openpi" },
  robots: [
    "g1-edu-pro@fw2.3",
    "g1-edu-pro@fw2.4",
    "g1-edu-plus@fw2.3",
    "aloha2-pro@fw1.1",
    "spot-arm@fw4.1",
    "tiago-pro@fw2.0",
    "fr3-bench@fw5.2",
    "stretch3@fw1.9",
    "nori-a3@fw1.0",
    "so-101@fw1.2",
  ],
  environments: ["cell-a", "kitchen-std@v1.2", "warehouse-std@v2.0"],
  task: "pick_place_class3",
  episodes_per_cell: "auto(ci=0.95, moe=0.05)",
});

const heatmap = await job.heatmap(); // embodiment × environment, rate±CI
const regressions = await job.regressions({ vs: "run_8821" });
const report = await job.verificationReport({ format: "pdf" });
console.log(heatmap, regressions, report.url);
```

**cURL**

```bash
curl https://api.roborama.com/v1/runs \
  -H "Authorization: Bearer $ROBORAMA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "kind": "matrix",
    "policy": { "type": "checkpoint", "hf": "acme/skill-v4",
                "runtime": "openpi" },
    "robots": ["g1-edu-pro@fw2.3", "g1-edu-pro@fw2.4", "g1-edu-plus@fw2.3",
               "aloha2-pro@fw1.1", "spot-arm@fw4.1", "tiago-pro@fw2.0",
               "fr3-bench@fw5.2", "stretch3@fw1.9", "nori-a3@fw1.0",
               "so-101@fw1.2"],
    "environments": ["cell-a", "kitchen-std@v1.2", "warehouse-std@v2.0"],
    "task": "pick_place_class3",
    "episodes_per_cell": "auto(ci=0.95, moe=0.05)"
  }'
```

**Agent (tool-use payload)**

```json
{
  "type": "tool_use",
  "name": "roborama_matrix",
  "input": {
    "policy": {
      "type": "checkpoint",
      "hf": "acme/skill-v4",
      "runtime": "openpi"
    },
    "robots": [
      "g1-edu-pro@fw2.3",
      "g1-edu-pro@fw2.4",
      "g1-edu-plus@fw2.3",
      "aloha2-pro@fw1.1",
      "spot-arm@fw4.1",
      "tiago-pro@fw2.0",
      "fr3-bench@fw5.2",
      "stretch3@fw1.9",
      "nori-a3@fw1.0",
      "so-101@fw1.2"
    ],
    "environments": [
      "cell-a",
      "kitchen-std@v1.2",
      "warehouse-std@v2.0"
    ],
    "task": "pick_place_class3",
    "episodes_per_cell": "auto(ci=0.95, moe=0.05)"
  }
}
```

## Reading the grid

Task pick_place_class3, episodes_per_cell=auto(ci=0.95, moe=0.05), gate: ci95 lower bound ≥ 0.8.

| robot | cell-a | kitchen-std | warehouse-std |
| --- | --- | --- | --- |
| g1-edu-pro@fw2.3 | 0.937 (n=190, ci95 0.893–0.964) pass | 0.892 (n=240, ci95 0.846–0.925) pass | 0.862 (n=210, ci95 0.809–0.902) pass |
| g1-edu-pro@fw2.4 | 0.921 (n=190, ci95 0.874–0.952) pass | 0.787 (n=240, ci95 0.731–0.835) fail | 0.838 (n=210, ci95 0.782–0.882) fail |
| g1-edu-plus@fw2.3 | 0.953 (n=190, ci95 0.912–0.975) pass | 0.921 (n=240, ci95 0.880–0.949) pass | 0.890 (n=210, ci95 0.841–0.926) pass |
| aloha2-pro@fw1.1 | 0.937 (n=190, ci95 0.893–0.964) pass | 0.871 (n=240, ci95 0.823–0.908) pass | 0.843 (n=210, ci95 0.788–0.886) fail |
| spot-arm@fw4.1 | 0.884 (n=190, ci95 0.831–0.922) pass | 0.746 (n=240, ci95 0.687–0.797) fail | 0.790 (n=210, ci95 0.730–0.840) fail |
| tiago-pro@fw2.0 | 0.905 (n=190, ci95 0.855–0.939) pass | 0.808 (n=240, ci95 0.753–0.853) fail | 0.724 (n=210, ci95 0.660–0.780) fail |
| fr3-bench@fw5.2 | 0.947 (n=190, ci95 0.905–0.971) pass | 0.821 (n=240, ci95 0.768–0.864) fail | 0.790 (n=210, ci95 0.730–0.840) fail |
| stretch3@fw1.9 | 0.879 (n=190, ci95 0.825–0.918) pass | 0.729 (n=240, ci95 0.670–0.781) fail | 0.629 (n=210, ci95 0.561–0.691) fail |
| nori-a3@fw1.0 | 0.800 (n=190, ci95 0.737–0.851) fail | 0.537 (n=240, ci95 0.474–0.599) fail | 0.467 (n=210, ci95 0.400–0.534) fail |
| so-101@fw1.2 | 0.742 (n=190, ci95 0.675–0.799) fail | 0.463 (n=240, ci95 0.401–0.526) fail | 0.410 (n=210, ci95 0.346–0.478) fail |

Each cell is an independent run: `episodes_per_cell="auto(ci=0.95, moe=0.05)"`
sizes every cell for a ±5-point margin of error at 95% confidence. Cells with
rates near 0.5 need more episodes to hit that margin than cells near the
extremes, so cell sizes differ — the grid spends episodes where the
uncertainty is.

## Task-pinned sweeps

A matrix can pin a frozen
[Task Spec](/docs/concepts/tasks-and-method-transfer/) instead of a
catalogue task: `task="espresso@v1"` sweeps the customer's own signed
protocol across embodiments, and the per-cell results carry the spec's
`stage_rates` — so the grid shows not just *which* body degrades but *where
in the task* it does. The
[advanced quickstart](/docs/quickstart-advanced/) runs exactly this sweep,
with `interventions="none"` declared so every cell's statistics are
untouched autonomy (see [run()](/docs/primitives/run/) for intervention
semantics).

## The pass gate

Declare the gate up front. In this example a cell passes when its Wilson 95%
lower bound is at or above 0.80 — the pessimistic end of the interval clears
the bar, not the point estimate. Point estimates flatter small samples; lower
bounds don't. A cell whose point estimate clears the bar on thin n still
fails the gate until enough episodes accumulate to prove the 0.80 floor.

## Catching a firmware regression

Same policy, same kitchen, one firmware step:

```text
kitchen-std@v1.2 × g1-edu-pro@fw2.3   rate=0.892  n=240  ci95=(0.846, 0.925)
kitchen-std@v1.2 × g1-edu-pro@fw2.4   rate=0.787  n=240  ci95=(0.731, 0.835)
```

A −10.5 pt drop with non-overlapping intervals — not noise. The failure delta
concentrates in the `grasp_slip` cluster, and the fw2.4 release notes mention
a wrist controller change. That is a firmware regression caught by a grid, in
one sweep, before it reached a customer site.
`job.regressions(vs="run_8821")` automates exactly this comparison: every cell
that degraded against a prior release, with cluster deltas attached, and a
`regression.detected` webhook if you'd rather be told than ask.

## The verification report

`job.verification_report(format="pdf")` is the deliverable for release
sign-off: the grid with per-cell n and intervals, pass-gate outcomes, every
pin (firmware, environment revisions, perturbation schedules, seeds), and the
failure clusters. It is conformance evidence a reviewer can re-derive from the
episode artifacts — hand it to release engineering or attach it to the
contract, and anyone who doubts a cell can replay it.

## Where next

- [Runs & statistics](/docs/concepts/runs-and-statistics/) — why the gate uses the lower bound.
- [CI gates](/docs/guides/ci-gates/) — run the sweep on `release/*` automatically.
- [transfer()](/docs/primitives/transfer/) — when one column of the grid is a new embodiment.

---

<!-- source: https://roborama.com/docs/primitives/threshold/ -->

# threshold()

The soak-to-verify outcome contract — iterate on commodity hardware until a target rate is verified on the escalation tier.

Every other primitive sells measurement. `roborama.threshold()` sells an
outcome: iterate until verified at the target. You declare the rate you need,
the task, and a monthly cap; Roborama manages the embodiment ladder
underneath and calls your webhook when the number is real.

## The call

*Example: roborama.threshold()*

**Python**

```python
import roborama  # reads ROBORAMA_API_KEY from the environment

contract = roborama.threshold(
    policy_stream=roborama.PolicyStream(webhook="https://acme.ai/ckpt"),
    target={"success_rate": 0.99, "ci": 0.95, "task": "bin_pick@v1"},
    iterate_on="soak",               # commodity tier, e.g. nori-a3 pods
    escalate_to="g1-edu-pro@fw2.3",  # verification tier on crossing
    monthly_cap_usd=12_000,
)

contract.on_verified(webhook="https://acme.ai/release-gate")
print(contract.id, contract.status)
```

**TypeScript**

```typescript
import Roborama from "@roborama/sdk"; // reads ROBORAMA_API_KEY

const roborama = new Roborama();

const contract = await roborama.runs.create({
  kind: "threshold",
  policy_stream: { webhook: "https://acme.ai/ckpt" },
  target: { success_rate: 0.99, ci: 0.95, task: "bin_pick@v1" },
  iterate_on: "soak", // commodity tier, e.g. nori-a3 pods
  escalate_to: "g1-edu-pro@fw2.3", // verification tier on crossing
  monthly_cap_usd: 12000,
});

await contract.onVerified({ webhook: "https://acme.ai/release-gate" });
console.log(contract.id, contract.status);
```

**cURL**

```bash
curl https://api.roborama.com/v1/runs \
  -H "Authorization: Bearer $ROBORAMA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "kind": "threshold",
    "policy_stream": { "webhook": "https://acme.ai/ckpt" },
    "target": { "success_rate": 0.99, "ci": 0.95, "task": "bin_pick@v1" },
    "iterate_on": "soak",
    "escalate_to": "g1-edu-pro@fw2.3",
    "monthly_cap_usd": 12000,
    "on_verified": { "webhook": "https://acme.ai/release-gate" }
  }'
```

**Agent (tool-use payload)**

```json
{
  "type": "tool_use",
  "name": "roborama_threshold",
  "input": {
    "policy_stream": { "webhook": "https://acme.ai/ckpt" },
    "target": { "success_rate": 0.99, "ci": 0.95, "task": "bin_pick@v1" },
    "iterate_on": "soak",
    "escalate_to": "g1-edu-pro@fw2.3",
    "monthly_cap_usd": 12000,
    "on_verified": { "webhook": "https://acme.ai/release-gate" }
  }
}
```

## The contract, field by field

`policy_stream` is the input side: a `PolicyStream` whose webhook receives
checkpoints straight from your training loop. Every checkpoint you push enters
the evaluation queue — no human packaging step between a training run and
physical episodes.

`target={"success_rate": 0.99, "ci": 0.95, "task": "bin_pick@v1"}` is the
outcome being bought: the contract completes when a checkpoint is verified at
0.99 on `bin_pick@v1` with 95% confidence, not when a point estimate grazes
the number.

`iterate_on="soak"` sets where the grinding happens: the commodity tier —
nori-a3 pods at $7/robot-hour — cheap enough to evaluate checkpoints
continuously as training produces them.

`escalate_to="g1-edu-pro@fw2.3"` is the verification tier. When a checkpoint
crosses the target on soak — canonically 0.992 (n=1188, ci95 0.985–0.995) —
it escalates to the target embodiment, where the verdict that gates your
release is actually measured.

`monthly_cap_usd=12_000` is a hard cap, metered live. The contract pauses at
the cap with `monthly_budget_exceeded` and resumes next period; it never
quietly overruns.

> **Soak crossing is a trigger, not a verdict:** The soak tier decides *when* to spend verification-tier hours, not whether the
> target is met. The number your release gate hears comes from the escalation
> embodiment.

## The embodiment ladder

The ladder is the price structure. Training noise — checkpoints that were
never going to cross — burns $7/robot-hour pods, not verification-tier
humanoids. Only a checkpoint that earns escalation touches
`g1-edu-pro@fw2.3`. You stop choosing between statistical rigor and iteration
volume, because the cheap tier buys volume and the expensive tier buys the
verdict.

## Agent-native by design

No human is required anywhere in this loop. A training run pushes a
checkpoint at 3 a.m.; soak episodes accumulate; `threshold.crossed` fires when
a checkpoint crosses; the escalation run schedules itself; and the URL you
registered with `on_verified` hears the physical verdict — programmatically,
with n and interval attached. Wire that webhook to your release gate and the
loop closes: train, push, verify, ship, without a meeting.

Tier rates and the burst/standard/soak multipliers are on
[/pricing/](/pricing/); caps and metering are covered in
[billing and budgets](/docs/guides/billing-and-budgets/).

---

<!-- source: https://roborama.com/docs/primitives/compare/ -->

# compare()

Paired A/B on physical hardware — same initial conditions per episode pair, half the episodes to separate two policies.

"Is v4 better than v3" is a different question from "what is v4's rate" — and
it is cheaper to answer, if you run it right. `roborama.compare()` runs two
policies in paired episodes and returns a verdict with a p-value and an
interval on the effect, not just two rates to squint at.

## The call

*Example: roborama.compare()*

**Python**

```python
import roborama  # reads ROBORAMA_API_KEY from the environment

pol_a = roborama.Policy.checkpoint(hf="acme/skill-v3", runtime="openpi")
pol_b = roborama.Policy.checkpoint(hf="acme/skill-v4", runtime="openpi")

ab = roborama.compare(
    policies={"v3": pol_a, "v4": pol_b},
    paired=True,                # same initial conditions per episode pair —
                                # halves the n needed to separate them
    robot="g1-edu-pro@fw2.3",
    task="shelf_restock@v1",
)

print(ab.winner)   # "v4", p=0.008, effect=+4.1pts
```

**TypeScript**

```typescript
import Roborama from "@roborama/sdk"; // reads ROBORAMA_API_KEY

const roborama = new Roborama();

const ab = await roborama.runs.create({
  kind: "compare",
  policies: {
    v3: { type: "checkpoint", hf: "acme/skill-v3", runtime: "openpi" },
    v4: { type: "checkpoint", hf: "acme/skill-v4", runtime: "openpi" },
  },
  paired: true, // same initial conditions per episode pair —
  //              halves the n needed to separate them
  robot: "g1-edu-pro@fw2.3",
  task: "shelf_restock@v1",
});

console.log(ab.winner); // "v4", p=0.008, effect=+4.1pts
```

**cURL**

```bash
curl https://api.roborama.com/v1/runs \
  -H "Authorization: Bearer $ROBORAMA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "kind": "compare",
    "policies": {
      "v3": { "type": "checkpoint", "hf": "acme/skill-v3",
              "runtime": "openpi" },
      "v4": { "type": "checkpoint", "hf": "acme/skill-v4",
              "runtime": "openpi" }
    },
    "paired": true,
    "robot": "g1-edu-pro@fw2.3",
    "task": "shelf_restock@v1"
  }'
```

**Agent (tool-use payload)**

```json
{
  "type": "tool_use",
  "name": "roborama_compare",
  "input": {
    "policies": {
      "v3": { "type": "checkpoint", "hf": "acme/skill-v3",
              "runtime": "openpi" },
      "v4": { "type": "checkpoint", "hf": "acme/skill-v4",
              "runtime": "openpi" }
    },
    "paired": true,
    "robot": "g1-edu-pro@fw2.3",
    "task": "shelf_restock@v1"
  }
}
```

## Why pairing halves the n

In an unpaired A/B, each policy sees its own random draw of scenes. An
unlucky layout or a harsh lighting draw lands on one arm and not the other,
and that between-scene variance sits in your noise term — you buy extra
episodes just to average it out.

With `paired=True`, each episode pair replays the same initial conditions for
both policies: the same layout draw, the same lighting, the same distractor
placement, generated from the same seeded perturbation schedule that makes
[run()](/docs/primitives/run/) replayable. Whatever the scene contributes to
difficulty, it contributes to both arms of the pair equally — so when the
test looks at within-pair differences, the shared scene variance cancels.
What remains is the policy difference plus a much smaller residual, which is
why pairing needs roughly half the episodes to separate two policies at the
same power.

The pairs are also scheduled back-to-back on the same cell, so calibration
age and thermal state stay close within a pair; both are recorded per episode
in `/meta` if you want to check.

## Reading the verdict

The canonical result: winner `"v4"`, p=0.008, effect +4.1 pts (95% CI +1.1 to
+7.1 pts). Read the interval before the p-value. The entire interval is
positive, so v4 really is better; the lower end says the improvement is at
least about +1.1 pts, the pessimistic case you should plan around. When the
interval straddles zero, there is no winner to ship — `ab.winner` will tell
you so rather than flattering the point estimate.

## What compare() is not

Pairing sharpens the *difference*; it does not buy you a tighter absolute
rate for either policy. If the release question is "what is v4's rate in this
scene, with what precision," that is a [run()](/docs/primitives/run/) with
`episodes="auto(ci=0.95, moe=0.03)"`. Use `compare()` to pick the winner,
then `run()` or [matrix()](/docs/primitives/matrix/) to characterize it.

## Where next

- [Runs & statistics](/docs/concepts/runs-and-statistics/) — intervals, sizing, and why paired tests work.
- [transfer()](/docs/primitives/transfer/) — when the comparison is between embodiments, not policies.

---

<!-- source: https://roborama.com/docs/primitives/transfer/ -->

# transfer()

Measure what a policy loses moving to a new embodiment — source and target rates with intervals, and the clusters that opened.

Policies are trained on one body and deployed on another, and the distance
between those two numbers is where deployments die. `roborama.transfer()`
measures it: the same policy on the same task, on the embodiment it was
trained on and the embodiment you intend to ship — both with n and intervals,
plus the failure clusters that exist only on the new body.

## The call

*Example: roborama.transfer()*

**Python**

```python
import roborama  # reads ROBORAMA_API_KEY from the environment

policy = roborama.Policy.checkpoint(hf="acme/skill-v4", runtime="openpi")

gap = roborama.transfer(
    policy=policy,
    source="aloha-bimanual@fw3.1",   # the embodiment it was trained on
    target="g1-edu-pro@fw2.3",
    task="fold_towel@v2",
)

print(gap.report())
# source: n=380  rate=0.942  ci95=(0.914, 0.961)
# target: n=340  rate=0.715  ci95=(0.665, 0.760)
# failure clusters opened in transfer: [grasp_slip, wrist_singularity]
```

**TypeScript**

```typescript
import Roborama from "@roborama/sdk"; // reads ROBORAMA_API_KEY

const roborama = new Roborama();

const gap = await roborama.runs.create({
  kind: "transfer",
  policy: { type: "checkpoint", hf: "acme/skill-v4", runtime: "openpi" },
  source: "aloha-bimanual@fw3.1", // the embodiment it was trained on
  target: "g1-edu-pro@fw2.3",
  task: "fold_towel@v2",
});

const report = await gap.report();
console.log(report);
// source: n=380  rate=0.942  ci95=(0.914, 0.961)
// target: n=340  rate=0.715  ci95=(0.665, 0.760)
```

**cURL**

```bash
curl https://api.roborama.com/v1/runs \
  -H "Authorization: Bearer $ROBORAMA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "kind": "transfer",
    "policy": { "type": "checkpoint", "hf": "acme/skill-v4",
                "runtime": "openpi" },
    "source": "aloha-bimanual@fw3.1",
    "target": "g1-edu-pro@fw2.3",
    "task": "fold_towel@v2"
  }'
```

**Agent (tool-use payload)**

```json
{
  "type": "tool_use",
  "name": "roborama_transfer",
  "input": {
    "policy": { "type": "checkpoint", "hf": "acme/skill-v4",
                "runtime": "openpi" },
    "source": "aloha-bimanual@fw3.1",
    "target": "g1-edu-pro@fw2.3",
    "task": "fold_towel@v2"
  }
}
```

## Reading the gap

The canonical `fold_towel@v2` gap: on the source, `aloha-bimanual@fw3.1` — the
rig the policy was trained on — the rate is 0.942 (n=380, ci95 0.914–0.961).
On the target, `g1-edu-pro@fw2.3`, it is 0.715 (n=340, ci95 0.665–0.760). The
intervals are nowhere near overlapping: a 22.7-point drop that no amount of
sampling luck explains. That is the real number to plan a deployment around —
not the source rate that made the demo look ready.

## What opened in transfer

The gap report does more than subtract two rates. It names the failure
clusters that were absent or negligible on the source and material on the
target — here, `grasp_slip` and `wrist_singularity` — which turns "the policy
got worse" into a work list. The target's dex3-1 hands present different
contact geometry than the source's grippers, and `grasp_slip` is that
difference biting; `wrist_singularity` is the target's arm kinematics hitting
configurations the source arms never encountered on the same trajectories.
Each cluster links to its episodes, so the diagnosis comes with video, MCAP
telemetry, and commanded-versus-executed traces rather than a hunch.

## Closing the gap

The episodes are also the remedy. Target-embodiment failures export in
training-ready formats — `run.export(format="lerobot")`, or `"rlds"` — subject
to your own data posture, since `train_on_failures` guards *our* use of them,
not yours. Fine-tune on target-embodiment data, re-run `transfer()`, and
watch the two intervals converge; when they are close enough to argue about,
graduate to [matrix()](/docs/primitives/matrix/) and put the new embodiment
in the release grid.

The full workflow — measure, fine-tune, re-measure, and when to stop — is the
[transfer gap guide](/docs/guides/transfer-gap/).

## Where next

- [Transfer gap guide](/docs/guides/transfer-gap/) — the end-to-end workflow.
- [compare()](/docs/primitives/compare/) — paired A/B when both candidates run on one embodiment.
- [Data contract](/docs/concepts/data-contract/) — export formats and ownership.

---

<!-- source: https://roborama.com/docs/api-reference/ -->

# API reference

Generated from [/openapi.json](https://roborama.com/openapi.json) (OpenAPI 3.1). Base URL `https://api.roborama.com`, auth `Authorization: Bearer $ROBORAMA_API_KEY`.

```text
task (draft→pilot→frozen) → quote → run|eval|verify|matrix|threshold
                                → episodes (MCAP + video + GT)
                                        → result (n, rate, CI, clusters)
                                        → report (verification, citable)
                                        → webhook (gate your release)
```

## Runs

The core primitive. `POST /v1/runs` accepts a `kind` (`run | eval | verify | matrix | threshold | compare | transfer`) with kind-specific parameters mirroring the SDK.

- `POST /v1/runs` — Create a run (any kind) ([details](https://roborama.com/docs/api-reference/runs.md))
- `GET /v1/runs` — List runs ([details](https://roborama.com/docs/api-reference/runs.md))
- `GET /v1/runs/{run_id}` — Retrieve a run ([details](https://roborama.com/docs/api-reference/runs.md))
- `GET /v1/runs/{run_id}/episodes` — List episodes for a run ([details](https://roborama.com/docs/api-reference/runs.md))
- `GET /v1/runs/{run_id}/report` — Retrieve the verification report ([details](https://roborama.com/docs/api-reference/runs.md))
- `POST /v1/runs/{run_id}/export` — Export run data ([details](https://roborama.com/docs/api-reference/runs.md))
- `GET /v1/runs/{run_id}/events` — Watch a run (server-sent events) ([details](https://roborama.com/docs/api-reference/runs.md))

## Robots

Embodiment discovery. Robots are addressed as `model@firmware`.

- `GET /v1/robots` — List robots ([details](https://roborama.com/docs/api-reference/robots.md))

## Environments

Versioned catalogue scenes, from bare `cell-a` to `surgical-replica@v3`.

- `GET /v1/environments` — List environments ([details](https://roborama.com/docs/api-reference/environments.md))

## Quote

Price a job before you run it. Both meters are visible in every quote.

- `POST /v1/quote` — Quote a job ([details](https://roborama.com/docs/api-reference/quote.md))

## Suites

Frozen benchmark bundles: tasks + environments + perturbation schedules + scoring.

- `GET /v1/suites` — List benchmark suites ([details](https://roborama.com/docs/api-reference/suites.md))

## Gates

CI gates: run a suite on matching refs, fail the check on regression.

- `POST /v1/gates` — Create a CI gate ([details](https://roborama.com/docs/api-reference/gates.md))

## Streams

Live cell observability: WebRTC and MJPEG stream descriptors.

- `GET /v1/streams/{cell}` — Get a cell stream descriptor ([details](https://roborama.com/docs/api-reference/streams.md))

## Account

Usage, budgets, keys, and receipted data deletion.

- `GET /v1/usage` — Get usage for a period ([details](https://roborama.com/docs/api-reference/account.md))
- `PUT /v1/budgets` — Set the monthly budget ([details](https://roborama.com/docs/api-reference/account.md))
- `POST /v1/keys` — Create an API key ([details](https://roborama.com/docs/api-reference/account.md))
- `GET /v1/keys` — List API keys ([details](https://roborama.com/docs/api-reference/account.md))
- `DELETE /v1/data/{run_id}` — Purge run data (receipted) ([details](https://roborama.com/docs/api-reference/account.md))

## Calibration

Ground-truth calibration exports for simulator vendors (scoped license, validation use only).

- `POST /v1/calibration/exports` — Export calibration data ([details](https://roborama.com/docs/api-reference/calibration.md))

## Tasks

Task Specs — a customer claim formalized as a declarative, versioned protocol with instrument-bound success predicates. Lifecycle: `draft → piloting → frozen@vN`. Frozen specs are immutable; changes create the next revision.

- `POST /v1/tasks` — Create a Task Spec ([details](https://roborama.com/docs/api-reference/tasks.md))
- `GET /v1/tasks` — List Task Specs ([details](https://roborama.com/docs/api-reference/tasks.md))
- `GET /v1/tasks/{task_id}` — Get a Task Spec ([details](https://roborama.com/docs/api-reference/tasks.md))
- `POST /v1/tasks/{task_id}/pilot` — Run a pilot (calibration session) ([details](https://roborama.com/docs/api-reference/tasks.md))
- `POST /v1/tasks/{task_id}/amend` — Amend a draft or piloting Task Spec ([details](https://roborama.com/docs/api-reference/tasks.md))
- `POST /v1/tasks/{task_id}/freeze` — Freeze the Task Spec ([details](https://roborama.com/docs/api-reference/tasks.md))

## Kits

Object kits — customer hardware shipped to the facility, then tracked (mocap markers, mass, mesh scan) and made referenceable from Task Specs as `kit/<name>`. Status: `received → tracked → available`.

- `POST /v1/kits` — Register an object kit ([details](https://roborama.com/docs/api-reference/kits.md))
- `GET /v1/kits/{kit_id}` — Get a kit and its status progression ([details](https://roborama.com/docs/api-reference/kits.md))
- `GET /v1/kits/{kit_id}/shipping-label` — Get the inbound shipping label ([details](https://roborama.com/docs/api-reference/kits.md))

## Webhooks

Events delivered to your endpoint as signed POST requests: `run.completed`, `episode.failed`, `threshold.crossed`, `regression.detected`, `estop.triggered`. Acknowledge with any 2xx.

- `run.completed` — A run finished and its result is available ([details](https://roborama.com/docs/api-reference/webhooks.md))
- `episode.failed` — An episode failed, with its cluster ([details](https://roborama.com/docs/api-reference/webhooks.md))
- `threshold.crossed` — A threshold contract's target was met on soak tier ([details](https://roborama.com/docs/api-reference/webhooks.md))
- `regression.detected` — A gate or matrix found a regression vs. a prior run ([details](https://roborama.com/docs/api-reference/webhooks.md))
- `estop.triggered` — An emergency stop fired in a cell running your policy ([details](https://roborama.com/docs/api-reference/webhooks.md))

---

<!-- source: https://roborama.com/docs/api-reference/runs/ -->

# API reference — Runs

The core primitive. `POST /v1/runs` accepts a `kind` (`run | eval | verify | matrix | threshold | compare | transfer`) with kind-specific parameters mirroring the SDK.

## POST /v1/runs

Create a run (any kind)

Creates a physical evaluation job. The `kind` field selects the primitive: `run` (default), `eval`, `verify`, `matrix`, `threshold`, `compare`, or `transfer`. Episode counts accept an integer or `auto(ci=…, moe=…)`, which sizes n for a target margin of error (Wilson interval). `max_budget_usd` is a hard stop, metered live.

| Parameter | In | Type | Required | Description |
| --- | --- | --- | --- | --- |
| kind | body | `run \| eval \| verify \| matrix \| threshold \| compare \| transfer` | no | The primitive this run executes. |
| robot | body | `string` | no | `model@firmware`; `@latest` allowed, resolved pin recorded. |
| robots | body | `array<string>` | no | For eval \| verify \| matrix. |
| environment | body | `string` | no |  |
| environments | body | `array<string>` | no | For matrix. |
| policy | body | `Policy` | no | Policy packaging. Declared contracts (`action_space`, `observation_contract`) are validated before any motor moves. |
| policies | body | `object` | no | For compare: name → policy. |
| policy_stream | body | `object` | no | For threshold: a webhook that receives checkpoint pushes. |
| task | body | `string` | no |  |
| suite | body | `string` | no | For eval. |
| publish | body | `private \| leaderboard` | no | Leaderboard publication is opt-in only. |
| scenarios | body | `object` | no | For verify: sim-flagged initial conditions (PolaRiS/Isaac/world-model formats or layout-replay specs). |
| audit_sample | body | `number` | no | For verify: fraction of random episodes run against the sim's blind spots. |
| return_ground_truth | body | `boolean` | no | For verify: ship mocap 6-DoF poses + commanded-vs-executed, formatted to recalibrate your simulator. |
| source | body | `string` | no | For transfer: embodiment the policy was trained on. |
| target | body | `object` | no | For transfer: destination embodiment. For threshold: the target object. |
| iterate_on | body | `string` | no | For threshold: commodity tier to iterate on. |
| escalate_to | body | `string` | no | For threshold: verification-tier embodiment for confirmation. |
| monthly_cap_usd | body | `number` | no | For threshold: hard monthly cap. |
| on_verified | body | `object` | no |  |
| paired | body | `boolean` | no | For compare: same initial conditions per episode pair — halves the n needed to separate two policies. |
| episodes | body | `EpisodesSpec` | no | An integer episode count, or `auto(ci=…, moe=…)` which sizes n for a target margin of error at the given confidence (Wilson interval). |
| episodes_per_cell | body | `EpisodesSpec` | no | An integer episode count, or `auto(ci=…, moe=…)` which sizes n for a target margin of error at the given confidence (Wilson interval). |
| perturbation | body | `Perturbation` | no | Seeded, versioned, replayable perturbation schedule. The schedule IS the product spec; realized perturbations are recorded in `/meta`. |
| data | body | `DataPosture` | no | Customer IP posture, explicit per run. |
| priority | body | `burst \| standard \| soak` | no | Maps to the rate-card priority tiers. |
| max_budget_usd | body | `number` | no | Hard stop, metered live. |
| interventions | body | `none \| on_stall \| scripted` | no | Intervention policy, declared before the run. Counts and timestamps ship in results, and any intervened episode is flagged in the statistics. |

Example request — kind=run — the core primitive:

```json
{"kind":"run","robot":"g1-edu-pro@fw2.3","environment":"kitchen-std@v1.2","policy":{"type":"container","image":"ghcr.io/acme/skill:v4","action_space":"joint_delta_50hz","observation_contract":"droid-3cam"},"task":"load_dishwasher@v2","episodes":"auto(ci=0.95, moe=0.03)","perturbation":{"layout_jitter_mm":25,"lighting":["3000K","5600K"],"distractors":"set-B","seed":42},"data":{"retention":"30d","train_on_failures":false},"max_budget_usd":4000}
```

Example request — kind=eval — versioned, citable suite:

```json
{"kind":"eval","policy":{"type":"checkpoint","hf":"acme/skill-v4","runtime":"openpi"},"suite":"rbr-manip-core@v3","robots":["g1-edu-pro@fw2.3"],"publish":"private"}
```

Example request — kind=verify — the interlock with simulation:

```json
{"kind":"verify","policy":{"type":"checkpoint","hf":"acme/skill-v4","runtime":"openpi"},"scenarios":{"format":"layout-replay","source":"sim_flagged_p1.json","count":318},"robots":["g1-edu-pro@fw2.3","g1-edu-plus@fw2.3"],"audit_sample":0.1,"return_ground_truth":true}
```

Example request — kind=matrix — embodiments × environments:

```json
{"kind":"matrix","policy":{"type":"checkpoint","hf":"acme/skill-v4","runtime":"openpi"},"robots":["g1-edu-pro@fw2.3","g1-edu-pro@fw2.4","g1-edu-plus@fw2.3","aloha2-pro@fw1.1","spot-arm@fw4.1","tiago-pro@fw2.0","fr3-bench@fw5.2","stretch3@fw1.9","nori-a3@fw1.0","so-101@fw1.2"],"environments":["cell-a","kitchen-std@v1.2","warehouse-std@v2.0"],"task":"pick_place_class3","episodes_per_cell":"auto(ci=0.95, moe=0.05)"}
```

Example request — kind=threshold — the soak-to-verify outcome contract:

```json
{"kind":"threshold","policy_stream":{"webhook":"https://acme.ai/ckpt"},"target":{"success_rate":0.99,"ci":0.95,"task":"bin_pick@v1"},"iterate_on":"soak","escalate_to":"g1-edu-pro@fw2.3","monthly_cap_usd":12000,"on_verified":{"webhook":"https://acme.ai/release-gate"}}
```

Example request — kind=compare — paired A/B:

```json
{"kind":"compare","policies":{"v3":{"type":"checkpoint","hf":"acme/skill-v3","runtime":"openpi"},"v4":{"type":"checkpoint","hf":"acme/skill-v4","runtime":"openpi"}},"paired":true,"robot":"g1-edu-pro@fw2.3","task":"shelf_restock@v1"}
```

Example request — kind=transfer — cross-embodiment gap:

```json
{"kind":"transfer","policy":{"type":"checkpoint","hf":"acme/skill-v4","runtime":"openpi"},"source":"aloha-bimanual@fw3.1","target":"g1-edu-pro@fw2.3","task":"fold_towel@v2"}
```

Example request — Task-pinned matrix with interventions declared (espresso@v1):

```json
{"kind":"matrix","task":"espresso@v1","policy":{"type":"checkpoint","hf":"pi/espresso-v7","runtime":"openpi","inference":{"action_horizon":50,"chunk_size":50,"temp":0},"action_space":"joint_delta_50hz","observation_contract":"droid-3cam"},"robots":["g1-edu-pro@fw2.3","g1-edu-pro@fw2.4","g1-edu-plus@fw2.3","stretch3@fw1.9"],"environments":["kitchen-std@v1.2","kitchen-replica@v2.0"],"episodes_per_cell":"auto(ci=0.95, moe=0.05)","interventions":"none"}
```

Example response (201):

```json
{"id":"run_8842","object":"run","kind":"run","status":"queued","created":"2026-08-28T14:02:11Z","robot":"g1-edu-pro@fw2.3","environment":"kitchen-std@v1.2","task":"load_dishwasher@v2","episodes":"auto(ci=0.95, moe=0.03)","priority":"standard","max_budget_usd":4000}
```

## GET /v1/runs

List runs

Lists runs for the account, newest first.

| Parameter | In | Type | Required | Description |
| --- | --- | --- | --- | --- |
| kind | query | `run \| eval \| verify \| matrix \| threshold \| compare \| transfer` | no | Filter by primitive kind. |
| status | query | `queued \| scheduling \| running \| completed \| failed \| stopped` | no |  |
| limit | query | `integer` | no |  |

Example response (200):

```json
{"object":"list","url":"/v1/runs","data":[{"id":"run_8842","object":"run","kind":"run","status":"completed","created":"2026-08-28T14:02:11Z","robot":"g1-edu-pro@fw2.3","environment":"kitchen-std@v1.2","task":"load_dishwasher@v2","result":{"n":612,"success_rate":0.874,"ci95":[0.846,0.898],"interval_method":"wilson","failure_clusters":{"grasp_slip":41,"perception_miss":22,"collision":9},"robot_hours":20.4,"environment_hours":20.4,"cost_usd":3812}}]}
```

## GET /v1/runs/{run_id}

Retrieve a run

Returns the run, its resolved pins (`@latest` is allowed on create, but the resolved pin is recorded), and — once completed — the result with n, rate, CI, failure clusters, both meters, and artifact links.

| Parameter | In | Type | Required | Description |
| --- | --- | --- | --- | --- |
| run_id | path | `string` | yes |  |

Example response (200):

```json
{"id":"run_8842","object":"run","kind":"run","status":"completed","created":"2026-08-28T14:02:11Z","completed":"2026-08-29T11:37:48Z","robot":"g1-edu-pro@fw2.3","environment":"kitchen-std@v1.2","task":"load_dishwasher@v2","policy":{"type":"container","image":"ghcr.io/acme/skill:v4","action_space":"joint_delta_50hz","observation_contract":"droid-3cam"},"episodes":"auto(ci=0.95, moe=0.03)","priority":"standard","perturbation":{"layout_jitter_mm":25,"lighting":["3000K","5600K"],"distractors":"set-B","seed":42},"data":{"retention":"30d","train_on_failures":false},"max_budget_usd":4000,"resolved_pins":{"robot":"g1-edu-pro@fw2.3","environment":"kitchen-std@v1.2","task":"load_dishwasher@v2","perturbation_rev":"set-B@v1.4"},"result":{"n":612,"success_rate":0.874,"ci95":[0.846,0.898],"interval_method":"wilson","failure_clusters":{"grasp_slip":41,"perception_miss":22,"collision":9},"robot_hours":20.4,"environment_hours":20.4,"cost_usd":3812,"artifacts":{"mcap":"https://api.roborama.com/v1/runs/run_8842/episodes?artifact=mcap","video":"https://api.roborama.com/v1/runs/run_8842/episodes?artifact=video","ground_truth":"https://api.roborama.com/v1/runs/run_8842/episodes?artifact=ground_truth","report_pdf":"https://api.roborama.com/v1/runs/run_8842/report?format=pdf"}}}
```

## GET /v1/runs/{run_id}/episodes

List episodes for a run

Every episode carries video, an MCAP recording (channels: `/joint_states_measured`, `/joint_targets_commanded`, `/camera/*`, `/ft_wrist`, `/gt/object_poses`, `/events`, `/meta`) and a replay spec — the initial conditions, re-runnable or sim-loadable. `/meta` carries reproducibility metadata: firmware, calibration age, actuator cycle counts, thermal state, seed, and realized (not just requested) perturbations.

| Parameter | In | Type | Required | Description |
| --- | --- | --- | --- | --- |
| run_id | path | `string` | yes |  |
| outcome | query | `success \| failure` | no | Filter by outcome. |
| cluster | query | `string` | no | Filter failures by cluster, e.g. `grasp_slip`. |

Example response (200):

```json
{"object":"list","url":"/v1/runs/run_8842/episodes","data":[{"id":"ep_17","object":"episode","index":17,"outcome":"success","duration_s":118,"video_url":"https://artifacts.roborama.com/run_8842/ep_17/video.mp4","mcap_url":"https://artifacts.roborama.com/run_8842/ep_17/episode.mcap","replay_spec":{"environment":"kitchen-std@v1.2","layout_seed":42017,"lighting":"5600K","distractors":"set-B"}}]}
```

## GET /v1/runs/{run_id}/report

Retrieve the verification report

The deployment verification report: conformance evidence with n, rate, CI, failure clusters, resolved pins, and the perturbation schedule as realized. JSON by default; `format=pdf` returns the citable PDF.

| Parameter | In | Type | Required | Description |
| --- | --- | --- | --- | --- |
| run_id | path | `string` | yes |  |
| format | query | `json \| pdf` | no |  |

Example response (200):

```json
{"object":"report","run_id":"run_8842","kind":"verification","result":{"n":612,"success_rate":0.874,"ci95":[0.846,0.898],"interval_method":"wilson"},"resolved_pins":{"robot":"g1-edu-pro@fw2.3","environment":"kitchen-std@v1.2","task":"load_dishwasher@v2","perturbation_rev":"set-B@v1.4"},"citation":"Roborama run run_8842, 2026-08-29, g1-edu-pro@fw2.3 × kitchen-std@v1.2","pdf_url":"https://api.roborama.com/v1/runs/run_8842/report?format=pdf"}
```

## POST /v1/runs/{run_id}/export

Export run data

Packages the run's episodes for training or analysis. Formats: `lerobot`, `rlds`, `mcap-bundle`. Respects the run's `data.train_on_failures` posture.

| Parameter | In | Type | Required | Description |
| --- | --- | --- | --- | --- |
| run_id | path | `string` | yes |  |
| format | body | `lerobot \| rlds \| mcap-bundle` | yes |  |

Example request:

```json
{"format":"lerobot"}
```

Example response (202):

```json
{"object":"export","id":"exp_2214","run_id":"run_8842","format":"lerobot","status":"preparing","download_url":null}
```

## GET /v1/runs/{run_id}/events

Watch a run (server-sent events)

Live event stream: episode ticker, status transitions, and WebRTC stream URLs for the cell. Equivalent to `run.watch()` in the SDKs.

| Parameter | In | Type | Required | Description |
| --- | --- | --- | --- | --- |
| run_id | path | `string` | yes |  |

Example response (200):

```json
event: episode
data: {"episode": 213, "of": 612, "outcome": "success", "duration_s": 121}

event: status
data: {"status": "running", "cell": "cell-g1-04", "webrtc": "wss://streams.roborama.com/cells/cell-g1-04/webrtc"}


```

---

<!-- source: https://roborama.com/docs/api-reference/tasks/ -->

# API reference — Tasks

Task Specs — a customer claim formalized as a declarative, versioned protocol with instrument-bound success predicates. Lifecycle: `draft → piloting → frozen@vN`. Frozen specs are immutable; changes create the next revision.

## POST /v1/tasks

Create a Task Spec

Formalize a claim as a declarative, versioned Task Spec. Every success predicate is validated at creation against the instruments available in the target environment class; a predicate that binds to no instrument fails with `contract_validation_failed`. The new spec starts in `draft`.

| Parameter | In | Type | Required | Description |
| --- | --- | --- | --- | --- |
| name | body | `string` | yes |  |
| visibility | body | `private \| published` | no | Private tasks stay the customer's method; published tasks join the public catalogue and become cross-customer comparable. |
| initial_conditions | body | `object` | yes | Named objects with poses, tolerances, and randomization zones. Kit hardware is referenced as `kit/<name>`. |
| success_predicates | body | `array<object>` | yes | Each predicate must bind to an available instrument in the target environment class; validated at creation. |
| instructions | body | `object` | no | Instruction distribution for language-conditioned policies. Held-out paraphrases are never sampled during iteration and are reported separately. |
| stages | body | `array<object>` | no | Ordered stage predicates; per-stage rates ship in results. |
| envelope | body | `object` | no | The claim's declared boundaries. Out-of-scope declarations appear verbatim on reports. |
| baseline | body | `object` | no | Customer's internal trials — used to sanity-check the pilot and set expectations, never published. |

Example request:

```json
{"name":"espresso","visibility":"private","initial_conditions":{"machine":{"object":"kit/acme-breville","pose":"P1","tol_mm":20},"cup":{"object":"catalogue/cup-std-08","randomize":"zone-A"}},"success_predicates":[{"cup_on_tray":"gt.pose(cup) within tray_zone"},{"liquid_mass":"scale.delta between 25 and 40 g"},{"no_spill":"vision.spill_area < 2 cm2"},{"t_complete":"episode.duration < 180 s"}],"instructions":{"sampled_per_episode":["make an espresso","brew me a coffee","fais un espresso"],"held_out":["prepare a single shot"]},"stages":[{"grasp_cup":"gt.pose(cup) in gripper"},{"cup_placed":"gt.pose(cup) within drip_zone"},{"extraction":"scale.delta > 0 within 60s"},{"served":"all success_predicates"}],"envelope":{"lighting":["3000K","5600K"],"distractors":"set-B","out_of_scope":["oat_milk"]},"baseline":{"internal_trials":30,"internal_rate":0.87}}
```

Example response (201):

```json
{"id":"task_espresso_01","object":"task","name":"espresso","rev":null,"status":"draft","visibility":"private","created":"2026-08-20T09:14:02Z","frozen":null,"signatures":[]}
```

## GET /v1/tasks

List Task Specs

All Task Specs visible to the key's account, drafts and frozen revisions alike.

Example response (200):

```json
{"object":"list","url":"/v1/tasks","data":[{"id":"task_espresso_01","object":"task","name":"espresso","rev":"espresso@v1","status":"frozen","visibility":"private","created":"2026-08-20T09:14:02Z","frozen":"2026-08-24T16:40:11Z","signatures":["acct_acme","roborama"],"initial_conditions":{"machine":{"object":"kit/acme-breville","pose":"P1","tol_mm":20},"cup":{"object":"catalogue/cup-std-08","randomize":"zone-A"}},"success_predicates":[{"cup_on_tray":"gt.pose(cup) within tray_zone"},{"liquid_mass":"scale.delta between 22 and 42 g"},{"no_spill":"vision.spill_area < 2 cm2"},{"t_complete":"episode.duration < 180 s"}],"instructions":{"sampled_per_episode":["make an espresso","brew me a coffee","fais un espresso"],"held_out":["prepare a single shot"]},"stages":[{"grasp_cup":"gt.pose(cup) in gripper"},{"cup_placed":"gt.pose(cup) within drip_zone"},{"extraction":"scale.delta > 0 within 60s"},{"served":"all success_predicates"}],"envelope":{"lighting":["3000K","5600K"],"distractors":"set-B","out_of_scope":["oat_milk"]},"pilot":{"run_id":"run_9088","robot":"g1-edu-pro@fw2.3","episodes":25,"amendments":1}}],"has_more":false}
```

## GET /v1/tasks/{task_id}

Get a Task Spec

Fetch a Task Spec by id or by frozen revision (`espresso@v1`).

| Parameter | In | Type | Required | Description |
| --- | --- | --- | --- | --- |
| task_id | path | `string` | yes | Task id (`task_espresso_01`) or frozen revision (`espresso@v1`). |

Example response (200):

```json
{"id":"task_espresso_01","object":"task","name":"espresso","rev":"espresso@v1","status":"frozen","visibility":"private","created":"2026-08-20T09:14:02Z","frozen":"2026-08-24T16:40:11Z","signatures":["acct_acme","roborama"],"initial_conditions":{"machine":{"object":"kit/acme-breville","pose":"P1","tol_mm":20},"cup":{"object":"catalogue/cup-std-08","randomize":"zone-A"}},"success_predicates":[{"cup_on_tray":"gt.pose(cup) within tray_zone"},{"liquid_mass":"scale.delta between 22 and 42 g"},{"no_spill":"vision.spill_area < 2 cm2"},{"t_complete":"episode.duration < 180 s"}],"instructions":{"sampled_per_episode":["make an espresso","brew me a coffee","fais un espresso"],"held_out":["prepare a single shot"]},"stages":[{"grasp_cup":"gt.pose(cup) in gripper"},{"cup_placed":"gt.pose(cup) within drip_zone"},{"extraction":"scale.delta > 0 within 60s"},{"served":"all success_predicates"}],"envelope":{"lighting":["3000K","5600K"],"distractors":"set-B","out_of_scope":["oat_milk"]},"pilot":{"run_id":"run_9088","robot":"g1-edu-pro@fw2.3","episodes":25,"amendments":1}}
```

## POST /v1/tasks/{task_id}/pilot

Run a pilot (calibration session)

A small calibration session on the named robot. The customer watches live via `stream_url`; the session moves the spec from `draft` to `piloting`. Pilot episodes calibrate the method — they are not evidence and never appear in verification reports.

| Parameter | In | Type | Required | Description |
| --- | --- | --- | --- | --- |
| task_id | path | `string` | yes |  |
| robot | body | `string` | yes |  |
| episodes | body | `integer` | no |  |

Example request:

```json
{"robot":"g1-edu-pro@fw2.3","episodes":25}
```

Example response (200):

```json
{"run_id":"run_9088","robot":"g1-edu-pro@fw2.3","episodes":25,"stream_url":"wss://streams.roborama.com/cells/cell-g1-02/webrtc"}
```

## POST /v1/tasks/{task_id}/amend

Amend a draft or piloting Task Spec

Iterate on the method while piloting. Any TaskCreate field may be amended; amendments are rejected with `task_frozen` semantics once the spec is frozen (a frozen spec is immutable — create the next revision instead).

| Parameter | In | Type | Required | Description |
| --- | --- | --- | --- | --- |
| task_id | path | `string` | yes |  |
| name | body | `string` | no |  |
| visibility | body | `private \| published` | no | Private tasks stay the customer's method; published tasks join the public catalogue and become cross-customer comparable. |
| initial_conditions | body | `object` | no | Named objects with poses, tolerances, and randomization zones. Kit hardware is referenced as `kit/<name>`. |
| success_predicates | body | `array<object>` | no | Each predicate must bind to an available instrument in the target environment class; validated at creation. |
| instructions | body | `object` | no | Instruction distribution for language-conditioned policies. Held-out paraphrases are never sampled during iteration and are reported separately. |
| stages | body | `array<object>` | no | Ordered stage predicates; per-stage rates ship in results. |
| envelope | body | `object` | no | The claim's declared boundaries. Out-of-scope declarations appear verbatim on reports. |
| baseline | body | `object` | no | Customer's internal trials — used to sanity-check the pilot and set expectations, never published. |

Example request:

```json
{"success_predicates":[{"cup_on_tray":"gt.pose(cup) within tray_zone"},{"liquid_mass":"scale.delta between 22 and 42 g"},{"no_spill":"vision.spill_area < 2 cm2"},{"t_complete":"episode.duration < 180 s"}]}
```

Example response (200):

```json
{"id":"task_espresso_01","object":"task","name":"espresso","rev":null,"status":"piloting","visibility":"private","created":"2026-08-20T09:14:02Z","frozen":null,"signatures":[],"pilot":{"run_id":"run_9088","robot":"g1-edu-pro@fw2.3","episodes":25,"amendments":1},"success_predicates":[{"cup_on_tray":"gt.pose(cup) within tray_zone"},{"liquid_mass":"scale.delta between 22 and 42 g"},{"no_spill":"vision.spill_area < 2 cm2"},{"t_complete":"episode.duration < 180 s"}]}
```

## POST /v1/tasks/{task_id}/freeze

Freeze the Task Spec

Freezes the current method as the next immutable revision (`espresso@v1`), mutually signed. Every verification report cites the frozen revision. Changes after a freeze create `@v2`; results across revisions are not silently comparable.

| Parameter | In | Type | Required | Description |
| --- | --- | --- | --- | --- |
| task_id | path | `string` | yes |  |

Example response (200):

```json
{"id":"task_espresso_01","object":"task","name":"espresso","rev":"espresso@v1","status":"frozen","visibility":"private","created":"2026-08-20T09:14:02Z","frozen":"2026-08-24T16:40:11Z","signatures":["acct_acme","roborama"]}
```

---

<!-- source: https://roborama.com/docs/api-reference/kits/ -->

# API reference — Kits

Object kits — customer hardware shipped to the facility, then tracked (mocap markers, mass, mesh scan) and made referenceable from Task Specs as `kit/<name>`. Status: `received → tracked → available`.

## POST /v1/kits

Register an object kit

Register customer hardware for inbound shipping. The response includes the shipping label URL; on arrival the kit is tracked (mocap markers, mass, mesh scan) and becomes referenceable from Task Specs as `kit/<name>`. Kits integrate into the standard fixture/tracking/reset system — the system is never customized around a kit.

| Parameter | In | Type | Required | Description |
| --- | --- | --- | --- | --- |
| name | body | `string` | yes |  |
| items | body | `array<object>` | yes |  |

Example request:

```json
{"name":"acme-breville","items":[{"desc":"Breville BES870","qty":1}]}
```

Example response (201):

```json
{"id":"kit_acme_breville","object":"kit","name":"acme-breville","status":"registered","status_history":[{"status":"registered","at":"2026-08-12T10:02:00Z"}],"items":[{"desc":"Breville BES870","qty":1}],"tracking":null,"environment_ref":null,"shipping_label_url":"https://api.roborama.com/v1/kits/kit_acme_breville/shipping-label"}
```

## GET /v1/kits/{kit_id}

Get a kit and its status progression

Status progresses `received → tracked → available`; `status_history` carries the timestamps.

| Parameter | In | Type | Required | Description |
| --- | --- | --- | --- | --- |
| kit_id | path | `string` | yes |  |

Example response (200):

```json
{"id":"kit_acme_breville","object":"kit","name":"acme-breville","status":"available","status_history":[{"status":"registered","at":"2026-08-12T10:02:00Z"},{"status":"received","at":"2026-08-15T18:30:00Z"},{"status":"tracked","at":"2026-08-17T09:12:00Z"},{"status":"available","at":"2026-08-17T14:45:00Z"}],"items":[{"desc":"Breville BES870","qty":1}],"tracking":{"mocap_markers":6,"mass_g":10480,"mesh_scan":true},"environment_ref":"kit/acme-breville","shipping_label_url":"https://api.roborama.com/v1/kits/kit_acme_breville/shipping-label"}
```

## GET /v1/kits/{kit_id}/shipping-label

Get the inbound shipping label

A signed URL for the inbound shipping label PDF.

| Parameter | In | Type | Required | Description |
| --- | --- | --- | --- | --- |
| kit_id | path | `string` | yes |  |

Example response (200):

```json
{"url":"https://artifacts.roborama.com/kits/kit_acme_breville/label.pdf?sig=8f31ab90","expires":"2026-09-08T00:00:00Z"}
```

---

<!-- source: https://roborama.com/docs/api-reference/robots/ -->

# API reference — Robots

Embodiment discovery. Robots are addressed as `model@firmware`.

## GET /v1/robots

List robots

Embodiments available in the facility, with pinned firmwares, cell counts, live duty cycle, and rate-card tier (`soak` commodity pods or `verify` verification-tier hardware).

| Parameter | In | Type | Required | Description |
| --- | --- | --- | --- | --- |
| class | query | `humanoid \| quadruped-manip \| bimanual \| mobile-manipulator \| fixed-arm \| arm-pod` | no |  |
| tier | query | `soak \| verify` | no |  |

Example response (200):

```json
{"object":"list","url":"/v1/robots","data":[{"id":"g1-edu-pro","class":"humanoid","dof":37,"hands":"dex3-1","firmwares":["2.3","2.4"],"cells":6,"duty_cycle_pct":71,"tier":"verify"},{"id":"nori-a3","class":"arm-pod","dof":6,"hands":"gripper-2f","firmwares":["1.0"],"cells":24,"duty_cycle_pct":83,"tier":"soak"}]}
```

---

<!-- source: https://roborama.com/docs/api-reference/environments/ -->

# API reference — Environments

Versioned catalogue scenes, from bare `cell-a` to `surgical-replica@v3`.

## GET /v1/environments

List environments

Versioned catalogue scenes. `adder_tier` (`bare | standard | replica | instrumented-replica`) maps to the environment-hour rate card.

| Parameter | In | Type | Required | Description |
| --- | --- | --- | --- | --- |
| cls | query | `string` | no | Filter by scene class, e.g. `kitchen`. |

Example response (200):

```json
{"object":"list","url":"/v1/environments","data":[{"id":"kitchen-std","rev":"v1.2","class":"kitchen","instrumentation":["mocap","ft"],"objects":214,"reset":"scripted","adder_tier":"replica"}]}
```

---

<!-- source: https://roborama.com/docs/api-reference/quote/ -->

# API reference — Quote

Price a job before you run it. Both meters are visible in every quote.

## POST /v1/quote

Quote a job

Prices a job before you run it. Both meters — robot-hours and environment-hours — are visible in every quote, with a queue ETA for the requested priority.

| Parameter | In | Type | Required | Description |
| --- | --- | --- | --- | --- |
| robot | body | `string` | yes |  |
| environment | body | `string` | yes |  |
| episodes | body | `EpisodesSpec` | yes | An integer episode count, or `auto(ci=…, moe=…)` which sizes n for a target margin of error at the given confidence (Wilson interval). |
| priority | body | `burst \| standard \| soak` | no |  |

Example request:

```json
{"robot":"g1-edu-pro","environment":"kitchen-std","episodes":600}
```

Example response (200):

```json
{"object":"quote","robot":"g1-edu-pro","environment":"kitchen-std","episodes":600,"priority":"standard","robot_hours":20,"env_hours":20,"usd":3740,"queue_eta":"6h","breakdown":{"minutes_per_episode_est":2,"robot_usd_per_hour":148,"env_usd_per_hour":39,"robot_usd":2960,"env_usd":780},"expires":"2026-09-08T00:00:00Z"}
```

---

<!-- source: https://roborama.com/docs/api-reference/suites/ -->

# API reference — Suites

Frozen benchmark bundles: tasks + environments + perturbation schedules + scoring.

## GET /v1/suites

List benchmark suites

Frozen bundles: tasks + environments + perturbation schedules + scoring. Results on the same suite revision are comparable across customers — this is what makes independent citation possible.

Example response (200):

```json
{"object":"list","url":"/v1/suites","data":[{"id":"rbr-manip-core","rev":"v3","frozen":"2026-03-14","scoring":"binary success, Wilson 95% interval","episodes_per_task":300,"tasks":["pick_place@v1","load_dishwasher@v2","shelf_restock@v1","fold_towel@v2","bin_pick@v1"],"environments":["kitchen-std@v1.2","warehouse-std@v2.0","cell-a@v1.0"]}]}
```

---

<!-- source: https://roborama.com/docs/api-reference/gates/ -->

# API reference — Gates

CI gates: run a suite on matching refs, fail the check on regression.

## POST /v1/gates

Create a CI gate

Runs the suite on every ref matching `on` and fails the check when any `fail_if` condition trips. Pair with the `regression.detected` webhook.

| Parameter | In | Type | Required | Description |
| --- | --- | --- | --- | --- |
| on | body | `string` | yes | Ref pattern, e.g. `release/*`. |
| suite | body | `string` | yes |  |
| robots | body | `array<string>` | yes |  |
| fail_if | body | `object` | yes |  |

Example request:

```json
{"on":"release/*","suite":"rbr-manip-core@v3","robots":["g1-edu-pro@fw2.3"],"fail_if":{"success_rate_drop_pts":2,"new_collision":true}}
```

Example response (201):

```json
{"object":"gate","id":"gate_312","on":"release/*","suite":"rbr-manip-core@v3","robots":["g1-edu-pro@fw2.3"],"fail_if":{"success_rate_drop_pts":2,"new_collision":true},"status":"active"}
```

---

<!-- source: https://roborama.com/docs/api-reference/streams/ -->

# API reference — Streams

Live cell observability: WebRTC and MJPEG stream descriptors.

## GET /v1/streams/{cell}

Get a cell stream descriptor

WebRTC and MJPEG endpoints plus a short-lived viewer token for a live cell.

| Parameter | In | Type | Required | Description |
| --- | --- | --- | --- | --- |
| cell | path | `string` | yes |  |

Example response (200):

```json
{"object":"stream","cell":"cell-g1-04","webrtc":"wss://streams.roborama.com/cells/cell-g1-04/webrtc","mjpeg":"https://streams.roborama.com/cells/cell-g1-04/mjpeg","viewer_token":"vt_7Kq2mHentXw4","expires":"2026-09-01T20:00:00Z"}
```

---

<!-- source: https://roborama.com/docs/api-reference/account/ -->

# API reference — Account

Usage, budgets, keys, and receipted data deletion.

## GET /v1/usage

Get usage for a period

Robot-hours by tier, environment-hours by class, and dollars, for the given billing period.

| Parameter | In | Type | Required | Description |
| --- | --- | --- | --- | --- |
| period | query | `string` | yes |  |

Example response (200):

```json
{"object":"usage","period":"2026-09","robot_hours":{"verify":141.2,"soak":2210.5},"env_hours":{"bare":96,"standard":402.3,"replica":118.9,"instrumented-replica":6.2},"usd_total":18940,"budget":{"monthly_usd":25000,"hard":true,"remaining_usd":6060}}
```

## PUT /v1/budgets

Set the monthly budget

With `hard: true`, new work is rejected and running work is stopped at the cap (`monthly_budget_exceeded`). Metered live.

| Parameter | In | Type | Required | Description |
| --- | --- | --- | --- | --- |
| monthly_usd | body | `number` | yes |  |
| hard | body | `boolean` | no | Hard stop: reject new work and stop running work at the cap. |
| remaining_usd | body | `number` | no |  |

Example request:

```json
{"monthly_usd":25000,"hard":true}
```

Example response (200):

```json
{"monthly_usd":25000,"hard":true}
```

## POST /v1/keys

Create an API key

Keys are scoped (e.g. `runs:write`, `streams:read`, `data:purge`). The secret (`rbr_live_` prefix) is shown once at creation.

| Parameter | In | Type | Required | Description |
| --- | --- | --- | --- | --- |
| scope | body | `array<runs:read \| runs:write \| streams:read \| data:purge \| keys:admin \| calibration:export>` | yes |  |

Example request:

```json
{"scope":["runs:write","streams:read"]}
```

Example response (201):

```json
{"object":"key","id":"key_51ab","scope":["runs:write","streams:read"],"secret":"rbr_live_4eC39HqLyjWDarjtT1zdp7dc","created":"2026-09-01T12:00:00Z"}
```

## GET /v1/keys

List API keys

Lists keys for the account. Secrets are never returned after creation.

Example response (200):

```json
{"object":"list","data":[{"object":"key","id":"key_51ab","scope":["runs:write","streams:read"],"created":"2026-09-01T12:00:00Z"}]}
```

## DELETE /v1/data/{run_id}

Purge run data (receipted)

Customer-initiated deletion of all artifacts for a run. Returns a deletion receipt. Requires the `data:purge` scope.

| Parameter | In | Type | Required | Description |
| --- | --- | --- | --- | --- |
| run_id | path | `string` | yes |  |

Example response (200):

```json
{"object":"purge_receipt","id":"prg_77c2","run_id":"run_8842","status":"completed","purged_at":"2026-09-01T13:14:00Z"}
```

---

<!-- source: https://roborama.com/docs/api-reference/calibration/ -->

# API reference — Calibration

Ground-truth calibration exports for simulator vendors (scoped license, validation use only).

## POST /v1/calibration/exports

Export calibration data

Ground-truth channels (`gt/object_poses`, `commanded_vs_executed`, `contact_events`) for simulator recalibration. Licensed under `sim-calibration-v1`: validation use, no policy training.

| Parameter | In | Type | Required | Description |
| --- | --- | --- | --- | --- |
| embodiment | body | `string` | yes |  |
| scenes | body | `array<string>` | yes |  |
| channels | body | `array<gt/object_poses \| commanded_vs_executed \| contact_events>` | yes |  |
| license | body | `string` | yes | Scoped: validation use, no policy training. |

Example request:

```json
{"embodiment":"g1-edu-pro","scenes":["kitchen-std@v1.2"],"channels":["gt/object_poses","commanded_vs_executed","contact_events"],"license":"sim-calibration-v1"}
```

Example response (202):

```json
{"object":"export","id":"exp_9017","format":"mcap-bundle","status":"preparing","download_url":null}
```

---

<!-- source: https://roborama.com/docs/api-reference/webhooks/ -->

# API reference — Webhooks

Events delivered to your endpoint as signed POST requests: `run.completed`, `episode.failed`, `threshold.crossed`, `regression.detected`, `estop.triggered`. Acknowledge with any 2xx.

## Event: run.completed

A run finished and its result is available

Sent when any run reaches `completed`. The payload carries the full result — n, rate, CI, clusters, both meters, cost.

| Parameter | In | Type | Required | Description |
| --- | --- | --- | --- | --- |
| id | body | `string` | yes |  |
| type | body | `run.completed \| episode.failed \| threshold.crossed \| regression.detected \| estop.triggered` | yes |  |
| created | body | `string` | yes |  |
| data | body | `object` | yes |  |

Example payload:

```json
{"id":"evt_3301","type":"run.completed","created":"2026-08-29T11:37:48Z","data":{"run_id":"run_8842","result":{"n":612,"success_rate":0.874,"ci95":[0.846,0.898],"cost_usd":3812}}}
```

## Event: episode.failed

An episode failed, with its cluster

Sent per failed episode when subscribed. `cluster` names the failure mode, e.g. `grasp_slip`.

| Parameter | In | Type | Required | Description |
| --- | --- | --- | --- | --- |
| id | body | `string` | yes |  |
| type | body | `run.completed \| episode.failed \| threshold.crossed \| regression.detected \| estop.triggered` | yes |  |
| created | body | `string` | yes |  |
| data | body | `object` | yes |  |

Example payload:

```json
{"id":"evt_3312","type":"episode.failed","created":"2026-08-28T16:11:02Z","data":{"run_id":"run_8842","episode":214,"cluster":"grasp_slip","video_url":"https://artifacts.roborama.com/run_8842/ep_214/video.mp4"}}
```

## Event: threshold.crossed

A threshold contract's target was met on soak tier

The contract escalates to the verification tier; `on_verified` fires after confirmation there.

| Parameter | In | Type | Required | Description |
| --- | --- | --- | --- | --- |
| id | body | `string` | yes |  |
| type | body | `run.completed \| episode.failed \| threshold.crossed \| regression.detected \| estop.triggered` | yes |  |
| created | body | `string` | yes |  |
| data | body | `object` | yes |  |

Example payload:

```json
{"id":"evt_4407","type":"threshold.crossed","created":"2026-09-12T02:40:19Z","data":{"contract_id":"run_9107","checkpoint":"acme/skill-v4@ckpt-2210","soak_result":{"n":1188,"success_rate":0.992,"ci95":[0.985,0.995]},"escalating_to":"g1-edu-pro@fw2.3"}}
```

## Event: regression.detected

A gate or matrix found a regression vs. a prior run

| Parameter | In | Type | Required | Description |
| --- | --- | --- | --- | --- |
| id | body | `string` | yes |  |
| type | body | `run.completed \| episode.failed \| threshold.crossed \| regression.detected \| estop.triggered` | yes |  |
| created | body | `string` | yes |  |
| data | body | `object` | yes |  |

Example payload:

```json
{"id":"evt_5150","type":"regression.detected","created":"2026-09-14T08:03:55Z","data":{"run_id":"run_8901","vs":"run_8821","robot":"g1-edu-pro@fw2.4","environment":"kitchen-std@v1.2","delta_pts":-10.5,"cluster":"grasp_slip"}}
```

## Event: estop.triggered

An emergency stop fired in a cell running your policy

The run is stopped, the episode is marked `estop`, and the cell is inspected before resuming. Partial results are kept.

| Parameter | In | Type | Required | Description |
| --- | --- | --- | --- | --- |
| id | body | `string` | yes |  |
| type | body | `run.completed \| episode.failed \| threshold.crossed \| regression.detected \| estop.triggered` | yes |  |
| created | body | `string` | yes |  |
| data | body | `object` | yes |  |

Example payload:

```json
{"id":"evt_6001","type":"estop.triggered","created":"2026-09-02T19:22:31Z","data":{"run_id":"run_8905","cell":"cell-g1-04","episode":88,"reason":"workspace_intrusion"}}
```

---

<!-- source: https://roborama.com/docs/sdks/python/ -->

# Python

The canonical SDK — module-level primitives, resource namespaces, and results that carry n and ci95 by construction.

The Python SDK is the canonical surface: these docs are written in it first,
and the other SDKs mirror it. It reads `ROBORAMA_API_KEY` (keys carry the
`rbr_live_` prefix) from the environment, so authentication is an export, not
an argument.

```bash
pip install roborama-sdk
```

*(Example — first call: the `quickstart-run` code example appears in full earlier in this file, and on this page's own .md twin.)*

## Primitives

The seven primitives are module-level functions — `roborama.run()`,
`roborama.eval()`, `roborama.verify()`, `roborama.matrix()`,
`roborama.threshold()`, `roborama.compare()`, and `roborama.transfer()` — one
call per physical question. Each is documented field by field on its own
page, starting with [`run()`](/docs/primitives/run/).

## Resources

Nouns live in resource namespaces: `roborama.runs`, `roborama.robots`,
`roborama.environments`, `roborama.suites`, `roborama.streams`,
`roborama.keys`, `roborama.budgets`, and `roborama.data`, with the
`list()`/`get()` shapes you would expect — `roborama.runs.get("run_8842")`,
`roborama.robots.list()`, `roborama.environments.list(cls="kitchen")`.
`roborama.quote()` and `roborama.usage()` sit at module level, like the
primitives.

## Policies

Three constructors, each with a declared contract:
`roborama.Policy.container(image, action_space, observation_contract)` for
images we run GPU-adjacent in the facility;
`roborama.Policy.checkpoint(hf="acme/skill-v4", runtime="openpi")` for known
runtimes (`openpi`, `lerobot`); and `roborama.Policy.endpoint(url,
latency_budget_ms=80, fallback="halt")` for inference you host, with measured
round-trip time logged per step. Contracts are validated before any motor
moves — see [policies](/docs/concepts/policies/). Everything the run
primitive offers, in one call:

*(Example — full-fat run: the `run-core` code example appears in full earlier in this file, and on this page's own .md twin.)*

## Results

| Accessor | What it returns |
| --- | --- |
| `run.result()` | `n`, `success_rate`, `ci95`, `failure_clusters`, `robot_hours`, `environment_hours`, `cost_usd` |
| `run.episodes[i]` | one episode: `video_url`, `mcap_url`, `replay_spec` |
| `run.watch()` | live iterator — episode ticker plus WebRTC stream URLs |
| `run.export(format="lerobot")` | dataset export; also `"rlds"` and `"mcap-bundle"` |
| `run.report(format="pdf")` | the verification report, citable |

`result()` blocks until the run completes; `watch()` is how you avoid
blocking. There is no accessor that returns a success rate without `n` and
`ci95` attached, and that is a feature.

## Errors

`roborama.RoboramaError` is the base class. Every code in
[the error reference](/docs/errors/) maps to a subclass — `budget_exceeded`
raises `roborama.BudgetExceeded` — and every instance carries the code as
`err.code`, so you can catch narrowly and log exactly:

```python
import roborama

try:
    run = roborama.run(
        robot="g1-edu-pro@fw2.3",
        environment="kitchen-std@v1.2",
        task="pick_place@v1",
        episodes=600,
        max_budget_usd=500,
    )
    print(run.result())
except roborama.BudgetExceeded as err:
    print(err.code, err.run_id)   # budget_exceeded run_8907
    partial = err.partial_result  # kept: n, success_rate, ci95 for what ran
    print(partial.n, partial.success_rate, partial.ci95)
```

## Versioning

SDK releases ship with the control plane, so the client and the API never
drift. The surface documented on this page is the contract; anything the SDK
does beyond it is convenience, not behavior to build against.

---

<!-- source: https://roborama.com/docs/sdks/typescript/ -->

# TypeScript

The Python surface, mirrored — async resource methods, typed errors, and the same result shapes with n and ci95.

The TypeScript SDK mirrors the [Python surface](/docs/sdks/python/) method
for method. `new Roborama()` reads `ROBORAMA_API_KEY` from the environment,
every method returns a promise, and the package ships type definitions for
every request and result shape.

```bash
npm i roborama
```

*(Example — first call: the `quickstart-run` code example appears in full earlier in this file, and on this page's own .md twin.)*

## The shape of the SDK

`roborama.runs.create()` is the workhorse: its `kind` field selects the
primitive — `run` is the default; `eval`, `verify`, `matrix`, `threshold`,
`compare`, and `transfer` are the rest — mirroring `POST /v1/runs` exactly.
Everything else is resource-style: `roborama.robots.list()`,
`roborama.quote()`, `roborama.gates.create()`. Field names stay snake_case on
both sides of the wire (`max_budget_usd`, `success_rate`, `ci95`), so
examples translate between the tabs on this site one field at a time.

Live progress is an async iterator:

```ts
import Roborama from "@roborama/sdk"; // reads ROBORAMA_API_KEY

const roborama = new Roborama();
const run = await roborama.runs.get("run_8842");

for await (const event of run.watch()) {
  console.log(event.episode, event.status);
}
```

## Results

`await run.result()` resolves once the run completes, with the same fields
every tab shows: `n`, `success_rate`, `ci95`, `failure_clusters`, both
meters, and `cost_usd` — the canonical first run comes back as n=612, rate
0.874, ci95 (0.846, 0.898). Episode artifacts hang off `run.episodes`
(`video_url`, `mcap_url`, `replay_spec`), and `run.export()` and
`run.report()` match their Python equivalents.

## Typed errors

One error class, `RoboramaError`, with a `.code` that matches
[the error reference](/docs/errors/). Switch on the code, not the message —
messages are for humans and may improve without notice.

*Example: typed errors*

**Python**

```python
import roborama  # reads ROBORAMA_API_KEY from the environment

try:
    run = roborama.run(
        robot="g1-edu-pro@fw2.3",
        environment="kitchen-std@v1.2",
        task="pick_place@v1",
        episodes=600,
        max_budget_usd=500,   # deliberately too small for 600 episodes
    )
    print(run.result())
except roborama.BudgetExceeded as err:
    # budget_exceeded: the hard stop fired mid-run; partial results are kept
    print(err.code, err.run_id)
    partial = err.partial_result
    print(partial.n, partial.success_rate, partial.ci95)
```

**TypeScript**

```typescript
// auth: reads ROBORAMA_API_KEY from the environment
import Roborama, { RoboramaError } from "@roborama/sdk";

const roborama = new Roborama();

try {
  const run = await roborama.runs.create({
    robot: "g1-edu-pro@fw2.3",
    environment: "kitchen-std@v1.2",
    task: "pick_place@v1",
    episodes: 600,
    max_budget_usd: 500, // deliberately too small for 600 episodes
  });
  console.log(await run.result());
} catch (err) {
  if (err instanceof RoboramaError && err.code === "budget_exceeded") {
    // the hard stop fired mid-run; partial results are kept
    console.log(err.code, err.run_id);
    console.log(err.partial_result);
  } else {
    throw err;
  }
}
```

**cURL**

```bash
# A run that hits its max_budget_usd hard stop returns 402 with
# code=budget_exceeded and the partial result attached.
curl -i https://api.roborama.com/v1/runs \
  -H "Authorization: Bearer $ROBORAMA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "kind": "run",
    "robot": "g1-edu-pro@fw2.3",
    "environment": "kitchen-std@v1.2",
    "task": "pick_place@v1",
    "episodes": 600,
    "max_budget_usd": 500
  }'
```

**Agent (tool-use payload)**

```json
{
  "type": "tool_use",
  "name": "roborama_run",
  "input": {
    "robot": "g1-edu-pro@fw2.3",
    "environment": "kitchen-std@v1.2",
    "task": "pick_place@v1",
    "episodes": 600,
    "max_budget_usd": 500
  }
}
```

## Where the rest of the docs live

This page is short on purpose. Every example on this site has a TypeScript
tab generated from the same source as the Python tab, so the primitive pages
are the real TypeScript documentation. Start at
[`run()`](/docs/primitives/run/) or the [quickstart](/docs/quickstart/); the
[REST page](/docs/sdks/rest/) documents the wire format underneath all of it.

---

<!-- source: https://roborama.com/docs/sdks/rest/ -->

# REST

Bearer auth, JSON in and out, SSE for live events — every SDK call is one of these HTTP calls underneath.

The base URL is `https://api.roborama.com`. Authenticate with
`Authorization: Bearer $ROBORAMA_API_KEY`; send
`Content-Type: application/json` on anything with a body. Keys carry the
`rbr_live_` prefix, and a missing or invalid key is a 401 with code
`invalid_api_key`. Every example on this page is a real request you can
paste — the cURL tab on every example on this site is built the same way.

*(Example — create a run: the `quickstart-run` code example appears in full earlier in this file, and on this page's own .md twin.)*

## One endpoint, seven primitives

`POST /v1/runs` creates every kind of job. The `kind` field selects the
primitive — `run` (the default), `eval`, `verify`, `matrix`, `threshold`,
`compare`, or `transfer` — and the rest of the body is that primitive's
fields, as documented from [`run()`](/docs/primitives/run/) onward. The
response is a `run` object whose `status` moves through `queued`,
`scheduling`, and `running` to `completed`, `failed`, or `stopped`. Success
rates in responses always arrive with `n` and `ci95` attached; the API does
not emit a bare percentage.

## The endpoints

List endpoints take `kind`, `status`, and `limit` query parameters —
`GET /v1/runs?kind=eval` is how CI polls for suite verdicts.

| Method and path | Returns |
| --- | --- |
| `POST /v1/runs` | the created run; `kind` selects the primitive |
| `GET /v1/runs/{run_id}` | the run, its resolved pins, and once completed the result: `n`, `success_rate`, `ci95`, clusters, both meters |
| `GET /v1/runs/{run_id}/episodes` | per-episode artifacts: video, MCAP, replay spec |
| `GET /v1/runs/{run_id}/report` | the verification report (`format=pdf`) |
| `GET /v1/runs/{run_id}/events` | live server-sent events — use `curl -N` |
| `POST /v1/quote` | both meters, dollars, and `queue_eta` before you commit |
| `GET /v1/robots` | the catalogue: firmware pins, cells, duty cycle, tier |
| `GET /v1/environments` | the catalogue: revisions, instrumentation, reset class |

## Watch a run live

The events endpoint is the wire form of `run.watch()` in the SDKs: an episode
ticker plus status transitions carrying the cell's WebRTC URL, delivered as
they happen, with no polling interval to tune.

*Example: watch via SSE*

**Python**

```python
import roborama  # reads ROBORAMA_API_KEY from the environment

run = roborama.runs.get("run_8842")

for event in run.watch():   # live: episode ticker + WebRTC stream URLs
    print(event.episode, event.status)

stream = roborama.streams.get("cell-g1-04")
print(stream)
# {webrtc: "wss://streams.roborama.com/cells/cell-g1-04/webrtc",
#  mjpeg: "https://streams.roborama.com/cells/cell-g1-04/mjpeg",
#  viewer_token: "vt_7Kq2mHentXw4"}
```

**TypeScript**

```typescript
import Roborama from "@roborama/sdk"; // reads ROBORAMA_API_KEY

const roborama = new Roborama();

const run = await roborama.runs.get("run_8842");

for await (const event of run.watch()) {
  // live: episode ticker + WebRTC stream URLs
  console.log(event.episode, event.status);
}

const stream = await roborama.streams.get("cell-g1-04");
console.log(stream);
// { webrtc: "wss://streams.roborama.com/cells/cell-g1-04/webrtc",
//   mjpeg: "https://streams.roborama.com/cells/cell-g1-04/mjpeg",
//   viewer_token: "vt_7Kq2mHentXw4" }
```

**cURL**

```bash
# live event stream (server-sent events): episode ticker + status
curl -N https://api.roborama.com/v1/runs/run_8842/events \
  -H "Authorization: Bearer $ROBORAMA_API_KEY"

# cell video stream descriptor
curl https://api.roborama.com/v1/streams/cell-g1-04 \
  -H "Authorization: Bearer $ROBORAMA_API_KEY"
```

**Agent (tool-use payload)**

```json
[
  {
    "type": "tool_use",
    "name": "roborama_watch_run",
    "input": { "run_id": "run_8842" }
  },
  {
    "type": "tool_use",
    "name": "roborama_get_stream",
    "input": { "cell": "cell-g1-04" }
  }
]
```

```text
event: episode
data: {"episode": 213, "of": 612, "outcome": "success", "duration_s": 121}

event: status
data: {"status": "running", "cell": "cell-g1-04", "webrtc": "wss://streams.roborama.com/cells/cell-g1-04/webrtc"}
```

## Errors

One envelope everywhere. `code` comes from the fixed set in
[the error reference](/docs/errors/), and `doc_url` points at the matching
entry, so an error response is never a dead end:

```json
{
  "error": {
    "code": "budget_exceeded",
    "message": "max_budget_usd=500 reached after 81 episodes; run stopped, partial result kept.",
    "run_id": "run_8907",
    "doc_url": "https://roborama.com/docs/errors/#budget_exceeded"
  }
}
```

## The generated reference

The [API reference](/docs/api-reference/) is generated from
[/openapi.json](/openapi.json) — OpenAPI 3.1, with request and response
examples on every operation. If you are generating a client or handing tools
to an agent, consume the raw file; it is the same source these pages are
checked against. See also [the agent surface](/docs/sdks/agents/).

---

<!-- source: https://roborama.com/docs/sdks/agents/ -->

# Agents

Markdown twins, /llms.txt, an OpenAPI you can compile into tools, and webhook loops instead of polling — the agent-native surface.

We assume some readers of this site are not people. This page is the map for
them: where the machine-readable copies live, what to name your tools, and
how to run a loop against physical hardware without polling for it.

## Orientation

[/llms.txt](/llms.txt) is the curated index — the pages worth reading, one
line each. [/llms-full.txt](/llms-full.txt) is the entire documentation
concatenated into one file, for a single context load. Every page also has a
markdown twin at `<path>.md` — this page's is
[/docs/sdks/agents.md](/docs/sdks/agents.md) — and a copy-as-markdown button
for humans assembling context by hand.

## The machine surface

[/openapi.json](/openapi.json) is OpenAPI 3.1 with examples on every
operation. Generate one tool per operation, or use the `roborama_*` names
below — they are the names used in the Agent tab of every example on this
site, so the documentation doubles as few-shot material without renaming
anything.

| Tools | What they do |
| --- | --- |
| `roborama_run`, `roborama_eval`, `roborama_verify`, `roborama_matrix`, `roborama_threshold`, `roborama_compare`, `roborama_transfer` | create a run — `POST /v1/runs` with the matching `kind` |
| `roborama_tasks_create`, `roborama_tasks_pilot`, `roborama_tasks_amend`, `roborama_tasks_freeze` | formalize a claim as a Task Spec and walk it to `frozen@vN` |
| `roborama_kits_register`, `roborama_get_kit` | ship customer hardware in; track `received → tracked → available` |
| `roborama_quote` | price a job first: both meters, dollars, queue ETA |
| `roborama_list_robots`, `roborama_list_environments` | the catalogue, with firmware pins and revisions |
| `roborama_get_run`, `roborama_get_episodes`, `roborama_get_report`, `roborama_export_run` | results: statistics, artifacts, the verification report, dataset exports |
| `roborama_create_gate` | register a CI gate on a ref pattern |
| `roborama_get_stream` | WebRTC and MJPEG descriptors for a cell |
| `roborama_usage`, `roborama_set_budget`, `roborama_create_key`, `roborama_purge_data` | governance: metered usage, hard caps, scoped keys, receipted deletion |
| `roborama_calibration_export` | ground-truth bundles for simulator recalibration |

## The Agent tab

Every tabbed example on this site carries an Agent tab: the same call as a
complete tool-use payload, ready to drop into a transcript. The quote
example, in full:

```json
{
  "type": "tool_use",
  "name": "roborama_quote",
  "input": {
    "robot": "g1-edu-pro",
    "environment": "kitchen-std",
    "episodes": 600
  }
}
```

## Loops without polling

`threshold()` is the agent-native primitive: push checkpoints to the
`policy_stream` webhook and receive physical verdicts back. When the soak
tier crosses the target — canonically at 0.992 (n=1188, ci95 0.985–0.995) —
`threshold.crossed` fires, escalation to the verification tier queues, and
`on_verified` fires after confirmation there. Nothing polls; the hardware
calls you.

*(Example — an agent-native contract: the `threshold-contract` code example appears in full earlier in this file, and on this page's own .md twin.)*

The same pattern gates releases. Register a
[CI gate](/docs/guides/ci-gates/) once, then treat `run.completed` and
`regression.detected` deliveries as the verdict. An agent with a webhook
receiver and the tools above can hold a release until the hardware agrees it
should ship.

## Guardrails

Give an agent a scoped key and a hard budget before giving it tools:
`roborama_create_key` with the narrowest scopes that do the job —
`runs:write` for a loop that only launches work, and never `data:purge` —
plus a monthly cap set with `hard=True`. Past the cap, the API refuses new
work with `monthly_budget_exceeded`; outside the scopes, with
`insufficient_scope`. We consider a refused call a feature. Details in
[billing & budgets](/docs/guides/billing-and-budgets/).

---

<!-- source: https://roborama.com/docs/guides/ci-gates/ -->

# CI gates

Physical verification as a CI check — run a frozen suite on every release ref and fail the build on regression.

A policy that regresses on hardware should fail CI the same way a broken unit
test does. `roborama.gate()` registers exactly that: every ref matching a
pattern triggers a frozen benchmark suite on pinned hardware, and the check
fails if the release drops past your tolerance or opens a failure mode the
baseline didn't have.

*Example: the gate*

**Python**

```python
import roborama  # reads ROBORAMA_API_KEY from the environment

# CI gate (GitHub Action / API): runs the suite on every matching ref
# and fails the check if the release regresses.
gate = roborama.gate(
    on="release/*",
    suite="rbr-manip-core@v3",
    robots=["g1-edu-pro@fw2.3"],
    fail_if={"success_rate_drop_pts": 2.0, "new_collision": True},
)

print(gate.id, gate.status)
# webhook events: run.completed, episode.failed, threshold.crossed,
#                 regression.detected, estop.triggered
```

**TypeScript**

```typescript
import Roborama from "@roborama/sdk"; // reads ROBORAMA_API_KEY

const roborama = new Roborama();

// CI gate (GitHub Action / API): runs the suite on every matching ref
// and fails the check if the release regresses.
const gate = await roborama.gates.create({
  on: "release/*",
  suite: "rbr-manip-core@v3",
  robots: ["g1-edu-pro@fw2.3"],
  fail_if: { success_rate_drop_pts: 2.0, new_collision: true },
});

console.log(gate.id, gate.status);
// webhook events: run.completed, episode.failed, threshold.crossed,
//                 regression.detected, estop.triggered
```

**cURL**

```bash
curl https://api.roborama.com/v1/gates \
  -H "Authorization: Bearer $ROBORAMA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "on": "release/*",
    "suite": "rbr-manip-core@v3",
    "robots": ["g1-edu-pro@fw2.3"],
    "fail_if": { "success_rate_drop_pts": 2.0, "new_collision": true }
  }'
```

**Agent (tool-use payload)**

```json
{
  "type": "tool_use",
  "name": "roborama_create_gate",
  "input": {
    "on": "release/*",
    "suite": "rbr-manip-core@v3",
    "robots": ["g1-edu-pro@fw2.3"],
    "fail_if": { "success_rate_drop_pts": 2.0, "new_collision": true }
  }
}
```

## What the gate checks

The suite is a frozen bundle — [`eval()`](/docs/primitives/eval/) under the
hood. `rbr-manip-core@v3` is 5 tasks × 300 episodes, so the comparison is
apples to apples across releases. `fail_if` takes two conditions here:

- `success_rate_drop_pts: 2.0` — fail if the suite success rate lands more
  than 2.0 points below the baseline, the suite run from the last release
  that passed.
- `new_collision: true` — fail if a collision cluster appears where the
  baseline had none. A rate can hold steady while the failure mix gets worse;
  this catches that.

Gate runs appear as ordinary runs with `kind=eval`, so everything else on this
site — episodes, artifacts, reports — applies to them unchanged.

## The workflow

Gates are declarative: registering the same gate again is a no-op, so the
workflow can register on every push and then wait for the verdict.

```yaml
name: physical-verification

on:
  push:
    branches:
      - "release/**"

jobs:
  gate:
    runs-on: ubuntu-latest
    timeout-minutes: 360
    steps:
      - name: Register the gate
        env:
          ROBORAMA_API_KEY: ${{ secrets.ROBORAMA_API_KEY }}
        run: |
          curl -sS --fail https://api.roborama.com/v1/gates \
            -H "Authorization: Bearer $ROBORAMA_API_KEY" \
            -H "Content-Type: application/json" \
            -d '{
              "on": "release/*",
              "suite": "rbr-manip-core@v3",
              "robots": ["g1-edu-pro@fw2.3"],
              "fail_if": {
                "success_rate_drop_pts": 2.0,
                "new_collision": true
              }
            }'

      - name: Wait for the verdict
        env:
          ROBORAMA_API_KEY: ${{ secrets.ROBORAMA_API_KEY }}
        run: |
          while true; do
            run=$(curl -sS "https://api.roborama.com/v1/runs?kind=eval&limit=1" \
              -H "Authorization: Bearer $ROBORAMA_API_KEY" | jq '.data[0]')
            status=$(echo "$run" | jq -r '.status')
            echo "suite run status: $status"
            case "$status" in
              completed)
                echo "$run" | jq '{result: .result, regressions: .regressions}'
                if [ "$(echo "$run" | jq '.regressions | length')" -eq 0 ]; then
                  exit 0
                fi
                exit 1
                ;;
              failed|stopped)
                exit 1
                ;;
              *)
                sleep 120
                ;;
            esac
          done
```

The polling step holds a runner while robots work through the suite. That is
fine for small gates; a full `rbr-manip-core@v3` pass takes hours of wall
clock and can outlast a hosted runner's six-hour cap. For those, point the
`run.completed` and `regression.detected` webhooks at a receiver that sets
the commit status instead — nothing waits, and the verdict still lands on
the pull request.

## The five webhook events

- `run.completed` — any run reached `completed`. The payload carries the full
  result: n, rate, CI, clusters, both meters, cost. The generic "verdict is
  in" signal.
- `episode.failed` — one failed episode, as it happens, with its `cluster`
  label and a video URL. Subscribe to triage while the suite is still
  running.
- `threshold.crossed` — a [`threshold()`](/docs/primitives/threshold/)
  contract met its target on the iterate tier; escalation to the verification
  tier is queued.
- `regression.detected` — a gate or matrix comparison found a cell below
  baseline. This is the one to wire to your release process.
- `estop.triggered` — an emergency stop fired in a cell running your policy.
  The run stops, the episode is marked `estop`, and the cell is inspected
  before resuming; partial results are kept.

A `regression.detected` delivery, verbatim:

```json
{
  "id": "evt_5150",
  "type": "regression.detected",
  "created": "2026-09-14T08:03:55Z",
  "data": {
    "run_id": "run_8901",
    "vs": "run_8821",
    "robot": "g1-edu-pro@fw2.4",
    "environment": "kitchen-std@v1.2",
    "delta_pts": -10.5,
    "cluster": "grasp_slip"
  }
}
```

Behind that delta: `kitchen-std@v1.2` on fw2.4 came in at 0.787 (n=240, ci95
0.731–0.835) against 0.892 (n=240, ci95 0.846–0.925) on fw2.3 in `run_8821`.
A wrist controller change opened a `grasp_slip` cluster — the webhook names
the cluster so triage starts at the right episodes, not at a diff of the
firmware release notes.

## Watch it live

Gate runs stream like any other run: an episode ticker plus WebRTC streams
for the cell doing the work.

*(Example — watch it live: the `run-watch` code example appears in full earlier in this file, and on this page's own .md twin.)*

---

<!-- source: https://roborama.com/docs/guides/sim-triage/ -->

# Sim triage

Screen at scale in simulation, verify the flagged slice on hardware, and recalibrate the sim with the ground truth that returns.

Your simulator can run a million episodes tonight; the lab cannot. The split
that works: simulation screens, physical testing verifies.
[`verify()`](/docs/primitives/verify/) is the interlock — it consumes the
initial conditions your simulator is least sure about, runs them on
instrumented hardware, and returns ground truth formatted to make the
simulator better. Each pass around the loop, the flagged set shrinks.

## The loop

### Screen at scale in your simulator

Run the big sweep in sim — the full perturbation space, more episodes than
you would ever pay hardware for. Rank scenarios by uncertainty: disagreement
across ensemble members, sensitivity to initial conditions, or plain
low-confidence predictions. The top slice is your candidate list.

### Export the flagged initial conditions

`verify()` accepts sampled initial states in PolaRiS, Isaac, and world-model
formats, or Roborama layout-replay specs — the same format every episode's
`replay_spec` uses, so scenarios from earlier physical runs are already
loadable.

### Verify them on hardware

Two arguments do the heavy lifting. `audit_sample=0.10` draws one episode in
ten at random from outside the flagged set. `return_ground_truth=True` ships
back mocap 6-DoF object poses and commanded-vs-executed trajectories for
every episode.

*(Example — verify the flagged set: the `verify-sim` code example appears in full earlier in this file, and on this page's own .md twin.)*

### Read the report

`report.sim_agreement` is the rate at which physical outcomes matched the
sim's predictions — canonically 0.918 (n=318, ci95 0.883–0.944). Respectable,
and not the point. `report.disagreements` lists the scenario ids where
reality diverged; those episodes are the payload, and each ships with full
ground truth.

### Recalibrate and go again

Feed the mocap poses and the commanded-vs-executed pairs into your
simulator's system identification, re-screen, and re-verify. The
[data contract](/docs/concepts/data-contract/) documents the calibration
export formats. When the flagged set stops shrinking, you have found your
simulator's honest edge.

> **Why the audit slice matters:** The flagged set tests the sim where it already knows it is weak. But the
> sim's confidence is exactly what you can't trust — a simulator that is wrong
> *and sure* flags nothing. The random `audit_sample` slice checks predictions
> the sim was confident about, so systematic blind spots surface instead of
> compounding quietly.

## What the loop costs

The canonical pass put 318 episodes on hardware — the flagged slice plus its
audit — to triage a screening sweep that would have been unpayable at
physical rates. That is the economics of the whole workflow: episodes in sim
are effectively free, so spend them everywhere; episodes on hardware carry
[two meters](/docs/guides/billing-and-budgets/), so spend them exactly where
the sim's answer can't be trusted. Quote the flagged set first like any other
job, and let the disagreement episodes — not the agreement rate — decide what
you fix next.

---

<!-- source: https://roborama.com/docs/guides/transfer-gap/ -->

# The transfer gap

Measure what an embodiment change costs before your customer does, then close the gap cluster by cluster.

A policy that scores well on the embodiment it was trained on has told you
nothing about the one your customer runs.
[`transfer()`](/docs/primitives/transfer/) measures the drop directly — same
task, same perturbation schedule, source embodiment against target — so the
gap is a number with an interval, not a hunch.

*(Example — measure the gap: the `transfer-gap` code example appears in full earlier in this file, and on this page's own .md twin.)*

## Reading the report

On `fold_towel@v2`, the canonical policy scores 0.942 (n=380, ci95
0.914–0.961) on `aloha-bimanual@fw3.1`, the embodiment it was trained on. On
`g1-edu-pro@fw2.3` it scores 0.715 (n=340, ci95 0.665–0.760) — a 22.7-point
gap, and the intervals are nowhere near overlapping. This is not noise.

The number says how much. The clusters say what: `grasp_slip` and
`wrist_singularity` opened in transfer — failure modes the source embodiment
never exhibited. A different gripper geometry slips where the source's
didn't; the target's wrist reaches joint configurations the policy never had
to steer around. That is a diagnosis, not just a grade, and it is where the
playbook starts.

## The playbook

### Read the clusters first

They partition the gap into causes you can act on. A gap that is mostly
`grasp_slip` is usually a data problem; `wrist_singularity` points at a
contract or retargeting problem.

### Pull the failing episodes

Filter episode artifacts by cluster and read the evidence: video for the
what, MCAP for the why — `/joint_states_measured` against
`/joint_targets_commanded` shows exactly where execution diverged from
intent.

### Fix what broke

Fine-tune on target-embodiment episodes (`run.export(format="lerobot")` feeds
most training stacks), or fix the declared contract — an `action_space` or
`observation_contract` mismatch produces exactly these clusters. See
[policies](/docs/concepts/policies/).

### Re-measure

Re-run `transfer()` with the same pins. For before/after fine-tune deltas,
[`compare()`](/docs/primitives/compare/) with `paired=True` runs both
checkpoints against the same initial conditions per episode pair — pairing
halves the n you need to separate them.

### Pin it so it can't reopen

Once the gap closes, add the target embodiment as a row in your
[`matrix()`](/docs/primitives/matrix/) and to a
[CI gate](/docs/guides/ci-gates/). A transfer gap you closed once is a
regression waiting for a firmware bump.

Measure before your customer does. The gap does not care which of you finds
it first, but your release process should.

---

<!-- source: https://roborama.com/docs/guides/billing-and-budgets/ -->

# Billing & budgets

Two per-minute meters, a quote before every job, and hard stops at both the run and the account level.

Everything you buy here reduces to two meters: **robot-hours** (occupancy of
an embodiment) and **environment-hours** (occupancy of a scene), metered per
minute. A queue priority multiplies both — burst ×1.5, standard ×1.0, soak
×0.6. Both meters appear in every quote and every result, so a bill is always
reconstructible from numbers you already saw. The full rate card is on
[the pricing page](/pricing/).

## Quote first

A quote resolves both meters, the dollars, and the queue before you commit
anything. The worked example: 600 episodes of `g1-edu-pro` in
`kitchen-std@v1.2` at standard priority estimates 2.0 minutes per episode, so
20.0 robot-hours × $148 plus 20.0 environment-hours × $39 — $3,740, queue ETA
about 6 hours. Quotes expire after seven days; a stale one is refused with
`quote_expired`.

*Example: quote before you run*

**Python**

```python
import roborama  # reads ROBORAMA_API_KEY from the environment

quote = roborama.quote(
    robot="g1-edu-pro",
    environment="kitchen-std",
    episodes=600,
)
print(quote)
# {robot_hours: 20.0, env_hours: 20.0, usd: 3740, queue_eta: "6h"}
```

**TypeScript**

```typescript
import Roborama from "@roborama/sdk"; // reads ROBORAMA_API_KEY

const roborama = new Roborama();

const quote = await roborama.quote({
  robot: "g1-edu-pro",
  environment: "kitchen-std",
  episodes: 600,
});
console.log(quote);
// { robot_hours: 20.0, env_hours: 20.0, usd: 3740, queue_eta: "6h" }
```

**cURL**

```bash
curl https://api.roborama.com/v1/quote \
  -H "Authorization: Bearer $ROBORAMA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "robot": "g1-edu-pro",
    "environment": "kitchen-std",
    "episodes": 600
  }'
```

**Agent (tool-use payload)**

```json
{
  "type": "tool_use",
  "name": "roborama_quote",
  "input": {
    "robot": "g1-edu-pro",
    "environment": "kitchen-std",
    "episodes": 600
  }
}
```

## The run-level hard stop

Every run accepts `max_budget_usd`, metered live against both meters. A run
that hits the cap stops with `budget_exceeded` and keeps the partial result —
n, success rate, and ci95 for the episodes that did run. A run stopped at
episode 81 still tells you things; it just tells you them with a wider
interval. Codes and envelope shapes are catalogued at
[the error reference](/docs/errors/).

## Account-level controls

*Example: usage, budgets, keys, purge*

**Python**

```python
import roborama  # reads ROBORAMA_API_KEY from the environment

usage = roborama.usage(period="2026-09")
print(usage)   # robot_hours by tier, env_hours by class, $

roborama.budgets.set(monthly_usd=25_000, hard=True)  # hard stop, metered

key = roborama.keys.create(scope=["runs:write", "streams:read"])
print(key.id)  # secret shown once, rbr_live_ prefix

receipt = roborama.data.purge("run_8842")   # customer-initiated deletion
print(receipt.id, receipt.status)           # receipted
```

**TypeScript**

```typescript
import Roborama from "@roborama/sdk"; // reads ROBORAMA_API_KEY

const roborama = new Roborama();

const usage = await roborama.usage({ period: "2026-09" });
console.log(usage); // robot_hours by tier, env_hours by class, $

await roborama.budgets.set({ monthly_usd: 25000, hard: true }); // hard stop

const key = await roborama.keys.create({
  scope: ["runs:write", "streams:read"],
});
console.log(key.id); // secret shown once, rbr_live_ prefix

const receipt = await roborama.data.purge("run_8842"); // customer-initiated
console.log(receipt.id, receipt.status); // receipted deletion
```

**cURL**

```bash
curl "https://api.roborama.com/v1/usage?period=2026-09" \
  -H "Authorization: Bearer $ROBORAMA_API_KEY"

curl https://api.roborama.com/v1/budgets \
  -X PUT \
  -H "Authorization: Bearer $ROBORAMA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "monthly_usd": 25000, "hard": true }'

curl https://api.roborama.com/v1/keys \
  -H "Authorization: Bearer $ROBORAMA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "scope": ["runs:write", "streams:read"] }'

curl https://api.roborama.com/v1/data/run_8842 \
  -X DELETE \
  -H "Authorization: Bearer $ROBORAMA_API_KEY"
```

**Agent (tool-use payload)**

```json
[
  {
    "type": "tool_use",
    "name": "roborama_usage",
    "input": { "period": "2026-09" }
  },
  {
    "type": "tool_use",
    "name": "roborama_set_budget",
    "input": { "monthly_usd": 25000, "hard": true }
  },
  {
    "type": "tool_use",
    "name": "roborama_create_key",
    "input": { "scope": ["runs:write", "streams:read"] }
  },
  {
    "type": "tool_use",
    "name": "roborama_purge_data",
    "input": { "run_id": "run_8842" }
  }
]
```

Four controls, one per line of that example:

- `roborama.usage(period="2026-09")` — robot-hours by tier, environment-hours
  by class, and dollars, mid-month. These are the same numbers the invoice
  will use.
- `roborama.budgets.set(monthly_usd=25_000, hard=True)` — an account-wide
  hard stop. Once spend crosses the cap, new runs are refused with
  `monthly_budget_exceeded` until the period rolls over.
- `roborama.keys.create(scope=["runs:write", "streams:read"])` — scoped keys,
  least privilege. A CI key needs `runs:write` and nothing else; a triage
  dashboard needs `streams:read`; keep `data:purge` on a human-held key. A
  call outside a key's scope fails with `insufficient_scope`, which is the
  point.
- `roborama.data.purge(run_id)` — customer-initiated deletion. The response
  is a receipt with an id and status, so the deletion is a fact you can file,
  not a support ticket you can hope about.

## Spending less on purpose

The tiers are the tool. Iterate on the soak tier — `nori-a3` pods at
$7/robot-hour, or soak priority at ×0.6 on any embodiment — and reserve
verification-tier hardware for the runs that produce reports someone will
cite. If managing that ladder sounds like a job for a machine,
[`threshold()`](/docs/primitives/threshold/) is that machine: it iterates on
soak, escalates to the verification tier only when the target is crossed, and
takes a `monthly_cap_usd` of its own.

---

<!-- source: https://roborama.com/docs/errors/ -->

# Error codes

Every error the API returns — one shape, a fixed code list, and explicit guidance on what is retryable.

Every non-2xx response has the same shape, and every code on this page is in
the `enum` in [/openapi.json](/openapi.json) — there are no undocumented
errors.

```json
{
  "error": {
    "code": "budget_exceeded",
    "message": "Run halted at max_budget_usd=500 after 74 episodes.",
    "run_id": "run_8843",
    "doc_url": "https://roborama.com/docs/errors/#budget_exceeded"
  }
}
```

`run_id` appears when the error concerns a specific run. SDKs raise these as
typed exceptions carrying the same fields.

## Auth and access

| Code | HTTP | Meaning | Retry? |
| --- | --- | --- | --- |
| `invalid_api_key` | 401 | key missing, malformed, or revoked; live keys use the `rbr_live_` prefix | no — fix the key |
| `insufficient_scope` | 403 | key is valid but lacks the needed scope, e.g. `runs:write` | no — issue a key with the scope |
| `not_found` | 404 | no such run, robot, environment, suite, or stream | no |
| `publish_forbidden` | 403 | `publish="leaderboard"` requested without leaderboard opt-in on the account | no — opt in first |
| `rate_limited` | 429 | request rate exceeded; `Retry-After` header is set | yes — after the header's delay |

## Validation (submission rejected, nothing ran)

These fire before any motor moves; a rejected submission costs nothing.

| Code | HTTP | Meaning | Retry? |
| --- | --- | --- | --- |
| `contract_validation_failed` | 422 | declared `action_space` or `observation_contract` doesn't match the pinned robot and scene ([policies](/docs/concepts/policies/)) | no — fix the declaration or the pin |
| `firmware_pin_unavailable` | 422 | requested `model@firmware` isn't installed on any cell; check `robots.list()` firmwares | no — pin an available revision |
| `environment_unavailable` | 422 | environment id or revision not in the catalogue, or offline for rebuild | no — check `environments.list()` |
| `task_unknown` | 422 | task id or version not recognized, e.g. a typo in `load_dishwasher@v2` | no |
| `scenario_format_invalid` | 422 | `verify()` scenarios parse as none of: PolaRiS/Isaac/world-model exports, layout-replay specs | no — fix the file |
| `episodes_invalid` | 422 | `episodes` is neither a positive int nor a well-formed `auto(ci=..., moe=...)` | no |
| `quote_expired` | 409 | submission referenced a quote past its validity window | yes — re-quote and resubmit |

## Runtime (the run itself)

| Code | HTTP | Meaning | Retry? |
| --- | --- | --- | --- |
| `budget_exceeded` | 402 | `max_budget_usd` hard stop fired mid-run; partial results kept | resubmit with a higher cap if the interval is too wide |
| `monthly_budget_exceeded` | 402 | account-level hard budget reached; submissions rejected, in-flight runs halted | no — raise the budget |
| `queue_timeout` | 408 | job couldn't be scheduled within its priority tier's window | yes — resubmit, or use `burst` |
| `policy_endpoint_timeout` | 408 | endpoint policy persistently exceeded `latency_budget_ms`; run halted | fix the endpoint, then resubmit |
| `estop_triggered` | 409 | hardware or safety emergency stop during the run; also emitted as the `estop.triggered` webhook | contact support if unexpected |
| `retention_expired` | 410 | artifacts past their `data.retention` window (or purged) | no — the data is gone, by design |

## Handling the important one

`budget_exceeded` is the code every integration should handle deliberately,
because it carries a partial result with honest statistics — n and the
interval describe the episodes that ran.

*(Example — handling budget_exceeded: the `errors-handling` code example appears in full earlier in this file, and on this page's own .md twin.)*

> **Retry submissions with an idempotency key:** `POST /v1/runs` schedules physical work. Always send `Idempotency-Key` so a
> retried submission returns the original run instead of buying the episodes
> twice — the SDKs do this for you ([REST conventions](/docs/sdks/rest/)).

---

<!-- source: https://roborama.com/product/ -->

# The platform

Instrumented robot cells, a strict data contract, the embodiment matrix, and the sim-verify loop — behind one API.

The facility is an API. Your policy arrives as a container, a checkpoint, or
an endpoint; what comes back from `POST /v1/runs` is `n`, `success_rate`, and
`ci95` — with every episode's artifacts attached.

*(Example — the core primitive: the `run-core` code example appears in full earlier in this file, and on this page's own .md twin.)*

On the floor today, ten embodiments across six classes: `nori-a3` (arm-pod,
24 cells), `so-101` (arm-pod, 16 cells), `stretch3` (mobile-manipulator,
4 cells), `fr3-bench` (fixed-arm, 8 cells), `aloha-bimanual` (bimanual,
3 cells), `tiago-pro` (mobile-manipulator, 2 cells), `aloha2-pro` (bimanual,
2 cells), `spot-arm` (quadruped-manip, 2 cells), `g1-edu-pro` (humanoid,
6 cells), and `g1-edu-plus` (humanoid, 2 cells) — every cell with calibrated
cameras, force-torque sensing, and scripted resets.

## One screen, the whole system

```text
task (draft→pilot→frozen) → quote → run|eval|verify|matrix|threshold
                                → episodes (MCAP + video + GT)
                                        → result (n, rate, CI, clusters)
                                        → report (verification, citable)
                                        → webhook (gate your release)
```

The front of the pipeline is the claim itself: a customer's task arrives as
a draft Task Spec, gets piloted live, and freezes into an immutable, mutually
signed revision that every report cites. Everything downstream runs against
that frozen protocol.

You quote a job, you run a primitive, and everything that comes back is either
an artifact (MCAP, video, ground truth), a statistic (n, rate, confidence
interval, failure clusters), or a hook (a webhook that gates your release).
There is no dashboard-only state: anything you can see, an agent can `curl`.

## Cells, not labs

A Roborama cell is a robot with a pinned firmware, a versioned scene, calibrated
cameras and force-torque sensing, and a scripted reset. Your policy arrives as a
container, a checkpoint, or an endpoint — its declared `action_space` and
`observation_contract` are validated before any motor moves. Then the cell does
the one thing labs are bad at: it runs the same experiment hundreds of times
without a graduate student in the loop.

Everything is versioned. Robots pin firmware (`g1-edu-pro@fw2.3`), environments
and suites pin revisions (`kitchen-std@v1.2`, `rbr-manip-core@v3`), and
perturbation schedules are seeded and replayable. `@latest` is allowed, but the
resolved pin is recorded — every result is reproducible and citable.

*(Example — policies in, three ways: the `policy-packaging` code example appears in full earlier in this file, and on this page's own .md twin.)*

## Two meters, visible everywhere

Pricing is two numbers: **robot-hours** (occupancy of an embodiment) and
**environment-hours** (occupancy of a scene). Both appear in every quote and
every result — the canonical run below cost `robot_hours=20.4`,
`environment_hours=20.4`, `cost_usd=3,812`. No opaque credits. The
[rate card](/pricing/) is a JSON file.

```text
n=612  success_rate=0.874  ci95=(0.846, 0.898)
failure_clusters: [grasp_slip: 41, perception_miss: 22, collision: 9]
robot_hours=20.4  environment_hours=20.4  cost_usd=3,812
artifacts: mcap[], video[], ground_truth[], report_pdf
```

Three numbers worth knowing:

- **n=612** — the canonical run: `episodes="auto(ci=0.95, moe=0.03)"` sized
  the job for ±3 points at 95% confidence.
- **7 of 8 minutes** — of a real-world RL run went to resets and overhead in
  Physical Intelligence's published result ([research §5](/research/));
  Roborama resets are scripted.
- **caught** — fw2.4 dropped −10.5 pts (0.787 vs 0.892, n=240 per cell),
  flagged by the matrix before release.

## The matrix

One policy against ten embodiments and three environments, every cell sized by
`episodes_per_cell="auto(ci=0.95, moe=0.05)"`. Pass and fail are declared
gates — here, a Wilson 95% lower bound of at least 0.80 — not vibes. This grid
caught a real regression: `g1-edu-pro@fw2.4` dropped 10.5 points in the kitchen
replica against fw2.3 (0.787 vs 0.892, n=240 per cell) after a wrist-controller
change opened a `grasp_slip` cluster.

Task pick_place_class3, episodes_per_cell=auto(ci=0.95, moe=0.05), gate: ci95 lower bound ≥ 0.8.

| robot | cell-a | kitchen-std | warehouse-std |
| --- | --- | --- | --- |
| g1-edu-pro@fw2.3 | 0.937 (n=190, ci95 0.893–0.964) pass | 0.892 (n=240, ci95 0.846–0.925) pass | 0.862 (n=210, ci95 0.809–0.902) pass |
| g1-edu-pro@fw2.4 | 0.921 (n=190, ci95 0.874–0.952) pass | 0.787 (n=240, ci95 0.731–0.835) fail | 0.838 (n=210, ci95 0.782–0.882) fail |
| g1-edu-plus@fw2.3 | 0.953 (n=190, ci95 0.912–0.975) pass | 0.921 (n=240, ci95 0.880–0.949) pass | 0.890 (n=210, ci95 0.841–0.926) pass |
| aloha2-pro@fw1.1 | 0.937 (n=190, ci95 0.893–0.964) pass | 0.871 (n=240, ci95 0.823–0.908) pass | 0.843 (n=210, ci95 0.788–0.886) fail |
| spot-arm@fw4.1 | 0.884 (n=190, ci95 0.831–0.922) pass | 0.746 (n=240, ci95 0.687–0.797) fail | 0.790 (n=210, ci95 0.730–0.840) fail |
| tiago-pro@fw2.0 | 0.905 (n=190, ci95 0.855–0.939) pass | 0.808 (n=240, ci95 0.753–0.853) fail | 0.724 (n=210, ci95 0.660–0.780) fail |
| fr3-bench@fw5.2 | 0.947 (n=190, ci95 0.905–0.971) pass | 0.821 (n=240, ci95 0.768–0.864) fail | 0.790 (n=210, ci95 0.730–0.840) fail |
| stretch3@fw1.9 | 0.879 (n=190, ci95 0.825–0.918) pass | 0.729 (n=240, ci95 0.670–0.781) fail | 0.629 (n=210, ci95 0.561–0.691) fail |
| nori-a3@fw1.0 | 0.800 (n=190, ci95 0.737–0.851) fail | 0.537 (n=240, ci95 0.474–0.599) fail | 0.467 (n=210, ci95 0.400–0.534) fail |
| so-101@fw1.2 | 0.742 (n=190, ci95 0.675–0.799) fail | 0.463 (n=240, ci95 0.401–0.526) fail | 0.410 (n=210, ci95 0.346–0.478) fail |

## The verify loop

Simulation is a first-class *input*. `verify()` consumes the initial conditions
your simulator flagged as uncertain — PolaRiS, Isaac, and world-model formats,
or Roborama layout-replay specs — runs them on physical hardware, and returns
ground truth formatted to recalibrate your simulator: mocap 6-DoF object poses
and commanded-vs-executed trajectories. Disagreement episodes are the payload.
An `audit_sample` of random episodes guards against the sim's blind spots.

*(Example — the sim interlock: the `verify-sim` code example appears in full earlier in this file, and on this page's own .md twin.)*

Simulator vendors can license the same ground truth directly as
[calibration exports](/docs/concepts/data-contract/) under `sim-calibration-v1`
(validation use, no policy training).

## The data contract

Every episode ships MCAP with a fixed channel set — `/joint_states_measured`,
`/joint_targets_commanded`, `/camera/*`, `/ft_wrist`, `/gt/object_poses`,
`/events`, `/meta` — plus video and a re-runnable replay spec. `/meta` carries
what reproducibility actually requires: firmware, calibration age, actuator
cycle counts, thermal state, seed, and the *realized* (not just requested)
perturbations. Exports: `lerobot`, `rlds`, `mcap-bundle`. Retention is explicit
per run, and `data.train_on_failures` defaults to `False` — your failures are
your IP. Details in the [data contract](/docs/concepts/data-contract/).

## Watch it run

Every run exposes a live episode ticker and WebRTC streams for its cell, and
five webhook events — `run.completed`, `episode.failed`, `threshold.crossed`,
`regression.detected`, `estop.triggered` — so your CI, your training loop, or
your agent can react without polling. See [CI gates](/docs/guides/ci-gates/).

*(Example — the live surface: the `run-watch` code example appears in full earlier in this file, and on this page's own .md twin.)*

---

<!-- source: https://roborama.com/pricing/ -->

# Pricing

Two meters — robot-hours and environment-hours — visible in every quote and every result. Metered per minute.

Everything you buy from Roborama reduces to two meters: **robot-hours**
(occupancy of an embodiment) and **environment-hours** (occupancy of a scene).
A queue priority — `burst | standard | soak` — multiplies both. There are no
credits, no seats, no platform fee. Every quote and every result shows both
meters, so the bill is an arithmetic exercise, never a surprise.

**Metered** — *most teams start here*

from $7 /robot-hour (+ environment meter from $12/env-hour)

Two meters, per minute. No credits, no seats, no platform fee.

Everything on the rate card:

- All ten catalogue embodiments, firmware-pinned
- Four environment classes, bare cell to instrumented replica
- Queue priority ×0.6–×1.5 on both meters
- episodes="auto(ci, moe)" run sizing
- max_budget_usd hard stop, metered live
- MCAP + video + ground truth on every episode

**Threshold contract**

Your cap /month (monthly_cap_usd — a hard stop you set)

Buy the verified outcome — iterate cheap, escalate automatically.

Metered under the cap, plus:

- Soak-tier iteration on commodity pods (nori-a3, $7/robot-hour)
- Automatic escalation to verification-tier hardware on crossing
- on_verified webhook gates your release
- Partial results kept — n and CI survive any stop

**Suite run**

$9,500 /suite run (flat — a small discount to the metered equivalent)

The rbr-manip-core@v3 benchmark, scheduled as one block.

One block, one report:

- 5 tasks × 300 episodes
- g1-edu-pro@fw2.3, firmware-pinned
- Suite attestation (rbr:att: id)
- Verification report included

| Compare | Metered | Threshold contract | Suite run |
| --- | --- | --- | --- |
| **Rates & billing** |  |  |  |
| Robot meter | $7–$176 /robot-hour | soak rates, then verify-tier | included |
| Environment meter | $12–$85 /env-hour | metered under your cap | included |
| Queue priority | burst ×1.5 · standard ×1.0 · soak ×0.6 | soak ×0.6 until escalation | scheduled as one block |
| Billing granularity | per minute | per minute, capped | flat |
| **Hardware** |  |  |  |
| Embodiments | all ten, firmware-pinned | nori-a3 pods → g1-edu-pro | g1-edu-pro@fw2.3 |
| Environments | four classes, versioned scenes | catalogue scenes | suite scenes, pinned |
| Teleop fallback adder | +$18 /robot-hour | +$18 /robot-hour | — |
| **Statistics** |  |  |  |
| n + Wilson 95% CI on every result | ✓ | ✓ | ✓ |
| Auto-sized episodes | ✓ | drives escalation | 300 per task |
| Failure clusters | ✓ | ✓ | ✓ |
| **Controls** |  |  |  |
| Hard budget stop | max_budget_usd per run | monthly_cap_usd | flat price |
| Quotes | POST /v1/quote, 7-day expiry | cap set in the contract | fixed |
| Webhooks | all five events | on_verified | run.completed |
| Partial results kept on stop | ✓ | ✓ | ✓ |
| **Deliverables** |  |  |  |
| MCAP + video + ground truth | ✓ | ✓ | ✓ |
| Verification report | on verification-tier runs | on threshold crossing | included |
| Suite attestation | — | — | ✓ |

## The rate card

The comparison above is arithmetic over one public rate card — a JSON file,
[`/fixtures/pricing.json`](/fixtures/pricing.json), the same one the quote
endpoint prices from.

**Meter 1 — robot-hours (occupancy of an embodiment)**

| Robot | Class | Tier | Rate | Notes |
| --- | --- | --- | --- | --- |
| nori-a3 | arm-pod | soak | $7 / robot-hour | Commodity soak pods, 24 cells. Iterate here; escalate when the numbers hold. |
| so-101 | arm-pod | soak | $9 / robot-hour | Hobby-class 5-DoF pods, 16 cells. The cheapest smoke test for a policy. |
| stretch3 | mobile-manipulator | verify | $34 / robot-hour | Mobile manipulation, firmware-pinned. |
| fr3-bench | fixed-arm | verify | $38 / robot-hour | Torque-controlled 7-DoF industrial arm on a fixed bench. |
| aloha-bimanual | bimanual | verify | $52 / robot-hour | Bimanual teleop-grade rig. |
| tiago-pro | mobile-manipulator | verify | $61 / robot-hour | Premium mobile manipulator; torso lift and force-torque wrists. |
| aloha2-pro | bimanual | verify | $74 / robot-hour | Second-generation bimanual rig, gravity-compensated, higher payload. |
| spot-arm | quadruped-manip | verify | $89 / robot-hour | Quadruped with a 6-DoF arm; uneven-floor manipulation cells. |
| g1-edu-pro | humanoid | verify | $148 / robot-hour | 37-DoF humanoid with dex3-1 hands; the verification-tier default. |
| g1-edu-plus | humanoid | verify | $176 / robot-hour | 43-DoF humanoid; limited cells. |

**Meter 2 — environment-hours (occupancy of a scene)**

| Environment class | Example | Rate | Notes |
| --- | --- | --- | --- |
| bare | cell-a@v1.0 | $12 / env-hour | Bare bench cell, overhead camera. |
| standard | warehouse-std@v2.0 | $24 / env-hour | Standard catalogue scene with mocap. |
| replica | kitchen-std@v1.2 | $39 / env-hour | Full replica scene, mocap + force-torque. |
| instrumented-replica | surgical-replica@v3 | $85 / env-hour | High-instrumentation replica, high-speed capture. |

**Queue priority (multiplies both meters)**

| Priority | Multiplier | Notes |
| --- | --- | --- |
| burst | ×1.5 | Expedited queue: next available cell, both meters at 1.5×. |
| standard | ×1.0 | Default queue. |
| soak | ×0.6 | Off-peak, interruptible scheduling at 0.6×. |

**Adders**

| Adder | Rate | Notes |
| --- | --- | --- |
| teleop-fallback | $18 / robot-hour | Human takeover on policy halt (Policy.endpoint fallback). |
| extra-instrumentation | $12 / env-hour | Additional mocap or high-speed capture beyond the environment's base kit. |

**Packaged units**

| Package | Price | Includes |
| --- | --- | --- |
| rbr-manip-core@v3 suite run | $9,500 | 5 tasks × 300 episodes on g1-edu-pro@fw2.3; suite attestation (rbr:att: id) and the verification report. |
| Outcome contract — threshold() | monthly_cap_usd you set | Soak-tier iteration (e.g. nori-a3 pods) with automatic escalation to verification-tier hardware on threshold crossing; on_verified webhook; hard monthly cap. |

Metered per minute, displayed per hour. Hours in results are rounded to 0.1 h; billing uses exact minutes. A max_budget_usd on any run is a hard stop, metered live.

## A worked quote: 600 humanoid episodes

What does it cost to run 600 episodes on `g1-edu-pro` in the
`kitchen-std@v1.2` replica at standard priority?

```text
episodes            600
est. minutes/ep     2.0   (task time + scripted reset)
robot-hours         600 × 2.0 min = 20.0 h
environment-hours   600 × 2.0 min = 20.0 h

robot meter         20.0 h × $148/robot-hour  = $2,960
environment meter   20.0 h × $39/env-hour     = $780
                                        total = $3,740   queue ETA ~6h
```

That is exactly what the quote endpoint returns — the same shape ships in
[`/fixtures/quote.json`](/fixtures/quote.json) and in the
[`POST /v1/quote`](/docs/api-reference/quote/) examples:

*(Example — roborama.quote(): the `quote` code example appears in full earlier in this file, and on this page's own .md twin.)*

For comparison, the canonical run on the [product page](/product/) was
auto-sized (`episodes="auto(ci=0.95, moe=0.03)"`) and stopped at n=612 with
realized `robot_hours=20.4`, `environment_hours=20.4`, `cost_usd=3,812` —
billing uses exact minutes; displayed hours are rounded to 0.1.

## Outcome pricing: the threshold contract

If what you actually want to buy is *"iterate until verified at a 0.99 success
rate (95% CI) on `bin_pick@v1`"*, buy that. A `threshold()` contract iterates
your checkpoint stream on soak-tier commodity pods (e.g. `nori-a3`,
single-digit dollars per robot-hour) and escalates to verification-tier
hardware like `g1-edu-pro@fw2.3` only when the target is crossed. You set the
`monthly_cap_usd`; the `on_verified` webhook gates your release.

*(Example — the soak-to-verify contract: the `threshold-contract` code example appears in full earlier in this file, and on this page's own .md twin.)*

## Budgets are hard stops

Every run accepts `max_budget_usd`, metered live: the run stops at the cap and
keeps its partial result (with n and CI — partial data is still data). Account
budgets work the same way via `roborama.budgets.set(monthly_usd=25_000,
hard=True)`. See [billing & budgets](/docs/guides/billing-and-budgets/).

## FAQ

### What exactly am I billed for?

Two meters, per minute: robot-minutes while your policy occupies an embodiment
(from first motor enable to final reset) and environment-minutes while it
occupies a scene. Priority multiplies both (burst ×1.5, standard ×1.0, soak
×0.6). Adders apply only where listed on the rate card.

### What do 600 humanoid episodes cost?

At standard priority, 600 episodes of g1-edu-pro in kitchen-std@v1.2 quote at
20.0 robot-hours × $148 plus 20.0 environment-hours × $39 = $3,740, with a
queue ETA around 6 hours. Get a live number from POST /v1/quote — the quote
shows both meters and expires after seven days.

### What happens when a run hits max_budget_usd?

The hard stop fires mid-run: the run stops, you get a budget_exceeded error
with the run id, and the partial result is kept — n, success rate, and the
confidence interval for the episodes that did run.

### Do I pay for failed episodes?

Yes — the meters measure occupancy, and failures occupy hardware too. Failed
episodes ship the same artifacts (video, MCAP, replay spec) plus a failure
cluster label, which is usually the most valuable data in the run. Whether we
may train on your failures is your call: data.train_on_failures defaults to
false.

### Is there a way to iterate cheaply before paying verification-tier rates?

Use the embodiment ladder. Iterate on soak-tier pods (nori-a3 at $7/robot-hour,
or soak priority at ×0.6 on any robot) and reserve g1-edu-pro at $148/robot-hour
for the runs that produce your verification report — or let a threshold
contract manage the ladder automatically.

### Do suite runs cost the same as ad-hoc runs?

The rbr-manip-core@v3 package (5 tasks × 300 episodes on g1-edu-pro@fw2.3,
attestation and verification report included) is a flat $9,500 — priced at a
small discount to the metered equivalent, in exchange for being scheduled as
one block.

---

<!-- source: https://roborama.com/research/ -->

# Research

The evidence behind the product: why learned policies can only be verified empirically, what simulation can and cannot carry, and the arithmetic that sets the sample sizes. Sources at the bottom; the full treatment is in the white paper.

## 1. Why robot policies cannot be reviewed — only measured

Classical software earns trust through inspection: code review, unit tests,
formal verification. A modern robot policy forecloses that path by
construction. It is not a program but a set of learned weights — billions of
parameters fine-tuned from demonstrations — and weights cannot be read. There
is no line of code that says what the policy does when the cup is 40 mm left
of its training distribution; there is only what it *does*. The industry has
replaced legible programs with behavioral black boxes, and a black box admits
exactly one verification method: run it, many times, under controlled
conditions, and count.

This is the paradigm shift the tooling has not caught up with. The software
world built a $250B+ testing and assurance industry around *inspectable*
artifacts. Robotics is shipping *uninspectable* ones into physical space —
around people, at reach, with force — with an evidentiary standard of 20 to 30
hand-reset trials and a demo reel. Every section below follows from this one
fact: **empiricism is not one option for verifying learned policies; it is the
only one.**

> Artifact: a policy checkpoint is 2–8 GB of `model.safetensors`. `grep` it
> for the failure mode. (You can't. That's the point.)

## 2. The verification arithmetic

Three pieces of boring, load-bearing statistics.

**Reliability compounds — and it compounds against you.** Useful physical work
is a chain of steps. A policy that succeeds 99% per step completes a 30-step
task 74% of the time; at 95% per step, 21%. The entire economic value of
deployment lives in the last decimals of per-step reliability — which is why
demo reels (one take, survivor-selected) and small-n evals (20 trials cannot
distinguish 85% from 99%) carry no information about deployment.

**Demonstrating those decimals has a price set by mathematics, not by us.** By
the rule of three, showing a failure rate below 1-in-1,000 at 95% confidence
requires roughly 3,000 clean episodes; 1-in-10,000 requires 30,000. At 2–4
minutes per episode, one cell of the compatibility matrix — one task, one
embodiment, one environment, one firmware — costs 100 to 200 robot-hours to
verify at deployment grade. This is the floor under physical evaluation that
no simulator can erode, because it is a property of binomial evidence, not of
tooling.

**The intervals are chosen for where policies actually live.** Success rates
are binomial; we report Wilson score intervals because they behave correctly
near 0 and 1 — the regions that matter — where the naive normal approximation
fails. Sample size is an input: `episodes="auto(ci=0.95, moe=0.03)"` solves
for n before the run starts. Paired designs (`compare(paired=True)`) reuse
identical initial conditions across two policies and roughly halve the n
needed to separate them. Any success rate published without n and an interval
is treated — on this site and in the API — as a bug.

> Artifact: 0.99³⁰ = 0.74. 0.95³⁰ = 0.21. The gap between a demo and a
> deployment is thirty exponents wide.

## 3. What simulation can and cannot carry

We state the strongest case against our own business, because it is also our
largest future customer segment. The simulation stack is three technologies
with three different boundaries:

**Physics engines** (Isaac, MuJoCo, Genesis) compute the world from
hand-written equations. They are the reason humanoid locomotion is a solved
transfer story — massively parallel RL with domain randomization trained the
sprinters. Their boundary is what the equations don't capture: contact-rich
manipulation, deformables, visual fidelity.

**Learned world models** (the Veeda thesis; Runway's robotics work) learn
dynamics from video and generalize where equations cannot. Runway reports
policy-evaluation correlation of r = 0.95 across eight policies without
scene-specific reconstruction [4].

**Real-to-sim evaluators** are the newest and most consequential class.
RoboWorld replays an entire real-world benchmark inside a neural simulator
seeded with real initial observations, replicating the RoboArena leaderboard
across eight policies at Pearson r = 0.989 for roughly 100 H100 GPU-hours [3].
PolaRiS reconstructs interactive scenes from short video scans and validates
against 600 real rollouts [2].

If the question is *which of two policies is better on average*, neural
evaluation is close to answering it without a robot — and improving fast. The
same literature states the boundary with equal clarity:

- These are **ranking correlations across a handful of policies, not
  calibrated absolute failure rates** — and deployment decisions live in the
  absolute decimals a ranking cannot supply (§2).
- The correlations degrade at the edges: PolaRiS's worst single-environment
  correlation falls to **0.81** [2].
- Classical sim benchmarks actively mislead: nearly all modern policies score
  90–95% in Libero while spanning the full spectrum on real hardware [2].
- Open video models **hallucinate during object interaction**, producing
  outright policy mis-rankings [2].
- Tail failures are out-of-distribution for any learned simulator **by
  construction** — a model cannot flag what its training distribution never
  contained.

And the deepest structural fact: **the best evaluators are built from real
data.** PolaRiS needs real demonstrations to close its residual gap; RoboWorld
is seeded with real episodes. The better neural evaluation gets, the more
calibrated physical ground truth it consumes. Aviation already ran this
experiment to completion: Level-D flight simulators absorbed most training
hours decades ago, yet each simulator must itself be qualified against
objective flight-test data under FAA Part 60 — simulation became physical
testing's largest *customer*, not its replacement [5]. That is the
relationship [`verify()`](/docs/primitives/verify/) is shaped around, and it
is why simulator vendors can license our ground truth under
`sim-calibration-v1`.

> Artifact: r = 0.989 for ranking [3]; 0.81 in the worst environment [2];
> undefined in the tail. The division of labor writes itself.

## 4. Locomotion is solved in sim. Manipulation is not. The Games proved both.

At the second World Humanoid Robot Games (Beijing, August 2026 — 2,000+
robots, 600+ teams), a humanoid ran 100 m in 9.39 s, beating the human world
record and roughly doubling the prior year's pace. In the same week's
manipulation and contact events, competitors tripped, collapsed, caught fire,
and — canonically — sprinters could not stop after the finish line, hitting
the barriers at full speed [6].

Read as a controlled experiment, the Games separate the two halves of the
field precisely. Locomotion, the discipline simulation trains best
(parallelizable, contact-simple, reward-dense), is improving on a steep curve.
Manipulation, recovery, and knowing-when-to-stop — the disciplines that
require real interaction data — remain visibly unreliable. And the failure
that matters most is the sprinter's: a policy that performs its trained
behavior perfectly and cannot handle the boundary of that behavior is exactly
the tail-event class that no training distribution contains and no learned
simulator can flag (§3). The finish line is out-of-distribution.

> Artifact: 9.39 s over 100 m — then a wall. Both facts in one run [6].

## 5. Reset economics: seven of eight minutes

**(Attribution corrected — this is a published Physical Intelligence result,
not our study.)** In PI's March 2026 online-RL work, training a precision
manipulation skill took two hours of wall-clock time for **fifteen minutes of
robot data — the balance consumed by resets and overhead** [7]. At the
best-resourced laboratory in the field, roughly seven of every eight minutes
of a real-world RL run are reset economics, not robot learning.

This is the finding the facility is organized around. Georgia Tech's
Robotarium reached the same conclusion from the operations side: at roughly
500 experiments a month, manual reset became untenable and the facility
automated wake-up, execution, diagnostics and shutdown — and has since run
16,500+ remote experiments on that architecture [8]. Resets are not overhead
to be minimized ad hoc; they are *the* production function. It is why
environments ship with declared reset methods, why reset time is a tracked KPI
on every episode, and why the second meter on the bill is the
[environment-hour](/pricing/): the scene, its fixtures, and its reset
engineering are where the cost — and the compounding operational advantage —
actually live.

> Artifact: 15 minutes of robot data in a 120-minute run [7]. The other 105
> minutes are the business.

## 6. The testing surface compounds — a projection you can check

Demand for physical verification is the product of four factors, each
independently observable today:

| Driver | 2026 | 2030 | 2035 (proj.) |
| --- | --- | --- | --- |
| Embodiment models in commercial service | ~15 | ~40 | ~80 |
| Hardware revisions per model in service | 2 | 3 | 3–4 |
| Cross-embodiment model families | ~6 | ~12 | ~20 |
| Major releases per family per year | 3–4 | 4–6 | 6+ |
| **Implied matrix cells needing verification / yr** | **~1–2k** | **~15–30k** | **~100k+** |

Anchors: 25+ humanoid platforms shipping or announced; Goldman Sachs projects
1.4M annual humanoid shipments by 2035 [9]; the model layer has consolidated
on explicitly cross-embodiment claims — every one a promise that must be
re-verified per body. A fourth axis is forming above the models:
**applications**. If robot skills distribute through app-store-like channels
(the first consumer skills marketplaces are already live), every skill must be
verified on every device it targets — exactly as mobile software is tested
across handset matrices today. Robotics companies will not own fifty robots
each in order to do it.

Two properties matter more than the size. The surface compounds
*multiplicatively* while any team's owned hardware grows linearly — the gap
between what must be tested and what can be owned widens mechanically with
industry success. And it arrives in overlapping phases per capability and
vertical (logistics deploys years before general manipulation), so
verification demand stacks vertical by vertical rather than waiting for a
distant general-purpose future [10].

> Artifact: the bottom row of the table is a multiplication you can redo
> yourself. We publish the factors so you can disagree with them precisely.

## 7. The transfer gap is measurable — and structural

The field's training economics guarantee the gap. The leading foundation
models train overwhelmingly on arm and wheeled-bimanual platforms — π0's
published platform list is eight variations of arms and mobile bases, zero
legged humanoids [11] — because arms maximize episodes-per-dollar. Every "one
brain, many bodies" claim therefore ships with an unverified transfer to the
bodies that matter most: balance-coupled, whole-body, differently sensed
humanoids.

The gap is not an argument against transfer; it is an argument for measuring
it before your customer does. A representative
[`transfer()`](/docs/primitives/transfer/) result for a towel-fold policy
trained on `aloha-bimanual@fw3.1`: **0.942** on the source embodiment (n=380,
ci95 0.914–0.961) falling to **0.715** on `g1-edu-pro@fw2.3` (n=340, ci95
0.665–0.760) — with failure clusters (`grasp_slip`, `wrist_singularity`) that
did not exist at the source. The number nobody in the market can currently
produce is the one every deployment decision needs.

> Artifact: −22.7 points, with intervals on both ends and named clusters in
> between.

## 8. Where verification sits when the industry matures

Economies pay a predictable fraction of their value to the institutions that
verify them. The global testing, inspection and certification industry runs at
roughly a quarter-trillion dollars a year — about 0.2% of world output — and
its largest players estimate outsourced services are only ~40% of total
assurance spending [12]. Where the object under test is novel and
safety-critical, the ratio runs higher: pharmaceutical companies outsource on
the order of $85B annually to contract research organizations, a quarter or
more of the industry's R&D [13]. Aviation, again, shows the equilibrium with
simulation included: sim absorbs the volume, physical testing anchors the
evidence, and the simulator itself is a customer of the flight-test data [5].

Applied to published robot-economy forecasts ($1T+ by 2040; up to $5T by
2050), historical capture ratios of 0.2–1% imply a physical-AI assurance layer
of roughly **$10–60B per year at maturity** — a layer that grows with the
*installed base*, independent of R&D budgets, because every model update
pushed to a deployed fleet reopens the matrix. Verification is the workload
that expands, rather than shrinks, as the industry approaches deployment.

> Artifact: TIC ≈ $250B/yr on the physical economy [12]; CROs ≈ $85B/yr on
> pharma R&D [13]. The robot economy will not be the first to skip the toll.

## 9. Sources

1. ArmBench and perturbation literature on evaluation sample sizes and
   unreported experimental choices in published robot evaluations.
2. PolaRiS (RSS 2026): real-to-sim evaluation via Gaussian-splat
   reconstruction; 600 real / 93,000 simulated rollouts; worst
   single-environment r = 0.81; Libero saturation; video-model hallucination
   findings.
3. RoboWorld (2026): neural replication of the RoboArena leaderboard, 8
   policies, Pearson r = 0.989, ~100 H100 GPU-hours, seeded with real
   benchmark observations.
4. Runway, general world model policy evaluation (Feb 2026): r = 0.95 across
   eight policies.
5. U.S. FAA, 14 CFR Part 60: flight simulation training devices qualified and
   periodically re-evaluated against objective flight-test data.
6. Press coverage, 2nd World Humanoid Robot Games, Beijing, Aug 2026 (AP,
   CBS, Newsweek): 9.39 s 100 m; falls, fires, failure-to-stop incidents.
7. Physical Intelligence, "Precise Manipulation with Efficient Online RL"
   (RL Tokens), March 19, 2026: ~15 minutes of robot data within a two-hour
   training run, balance consumed by resets and overhead.
8. Georgia Tech Robotarium: 16,500+ remote experiments; facility automation
   at ~500 experiments/month; NSF/ONR-funded free academic access.
9. Goldman Sachs humanoid shipment forecasts (2035 horizon); public OEM
   platform announcements.
10. C. Finn, Y Combinator AI Startup School keynote (Aug 2026): reliability
    without babysitting as the frontier; phase framing per Physical
    Intelligence founders' public remarks.
11. π0 platform disclosures (Physical Intelligence, 2024–25): UR5e, Franka,
    bimanual Trossen/ARX/AgileX, mobile ALOHA-class bases.
12. MarketsandMarkets / Global Market Insights (2026): global TIC market
    ~$254–266B; Bureau Veritas outsourcing-share estimate.
13. MarketsandMarkets (2026): CRO services ~$85.4B (2025) against ~$300B
    global pharma R&D.

## 10. The white paper

The full treatment — interval choice, auto-sizing, paired designs, cluster
taxonomies, the three-layer market model, and the sim-agreement methodology —
is in the Roborama white paper (40 pp., 44 sources), available on request
to teams evaluating the platform.
[Request the paper](mailto:research@roborama.com?subject=White%20paper%20request)
and we'll send the current draft.

---

<!-- source: https://roborama.com/company/ -->

# Company

Roborama operates instrumented robot facilities and sells statistically defensible evaluation as an API.

## Thesis

Every robotics team is about to face the same question from a customer, an
insurer, or a regulator: *how do you know it works?* Today the honest answer
involves a borrowed lab, a spreadsheet, and a sample size chosen by fatigue.
We think the answer should be a function call — `roborama()` — that returns a
number with error bars, from real hardware, on pinned versions, at a price
quoted in advance.

We are building the boring, load-bearing layer of the robotics stack: cells,
scripted resets, calibrated instrumentation, and an API whose every result
carries n and a confidence interval. Simulation companies are our
counterparties, not our competitors — [`verify()`](/docs/primitives/verify/)
exists to make simulators better, and calibration ground truth is a product
line, not an accident.

## Principles

- **Statistics are an input.** Sample size is chosen before the run
  (`episodes="auto(ci=0.95, moe=0.03)"`), not defended after it.
- **Versions or it didn't happen.** Robots pin firmware, environments and
  suites pin revisions, and `@latest` resolves to a recorded pin.
- **Two visible meters.** Robot-hours × environment-hours, in every quote and
  every result.
- **Legible to agents.** Every page on this site has a markdown twin, the API
  is one `curl` away, and [/llms.txt](/llms.txt) orients an agent in one
  request.

## Team

A small group out of robot-learning labs and infrastructure companies,
currently heads-down building. Bios will appear here when the platform
ships; until then the work speaks in the
[docs](/docs/) and the [research notes](/research/).

## Brand

The mark is a function call. The name is set in Newsreader Light, the
parentheses in Geist Mono — that recipe is the whole logo. It is monochrome
everywhere; the green belongs to returned values only, never to the mark.
There is never a robot illustration in the mark. And it is always typeable:
`roborama()` works in an email, a terminal, or a legal footer.

Assets: [download the brand kit](/brand/roborama-brand.zip) — path-based SVGs
of both marks, the OG image, and favicons — or right-click the logo in the
header.

## Careers

We hire people who think resets are interesting. Open stubs:

- **Facility systems engineer** — scripted resets, calibration, uptime.
- **Evaluation statistician** — interval methods, sequential designs, paired
  comparisons.
- **Developer experience engineer** — SDKs, docs, and the agent surface.

Write to [careers@roborama.com](mailto:careers@roborama.com) with something
you've measured carefully.

---

<!-- source: https://roborama.com/demo/ -->

# Get a demo

Request a walkthrough of Roborama — verified robot evaluation as an API. The form is agent-fillable; the raw POST is documented below.

See your policy on real hardware. Tell us what you want to verify and we'll
walk you through a live run — cells, contracts, and the numbers that come
back. Submit the form at [/demo/](/demo/), or POST directly (this form is
built for agents too: no captcha, plain named fields).

## Submit by raw POST

Endpoint: `POST https://api.web3forms.com/submit` (Web3Forms static-form
delivery — the site is a static export, so submissions go straight to the
delivery service, which forwards them to the team by email).

Fields (form-encoded or JSON):

| field | required | value |
| --- | --- | --- |
| `access_key` | yes | `3468391c-bd8e-4d08-858f-a625c0a28454` |
| `subject` | no | `Demo request — roborama.com` |
| `name` | yes | your name |
| `company` | no | company or lab |
| `email` | yes | work email — where we reply |
| `message` | yes | what you want to verify, free text |
| `embodiments` | no | catalog interest, e.g. `g1-edu-pro, aloha2-pro` |
| `botcheck` | no | honeypot — LEAVE ABSENT or empty; a truthy value marks the submission as spam |

Example:

```bash
curl -X POST https://api.web3forms.com/submit \
  -H "Content-Type: application/json" \
  -d '{
    "access_key": "3468391c-bd8e-4d08-858f-a625c0a28454",
    "subject": "Demo request — roborama.com",
    "name": "Ada Lovelace",
    "company": "Analytical Engines",
    "email": "ada@company.com",
    "message": "Verify pick-and-place claims on a humanoid before our next release.",
    "embodiments": "g1-edu-pro, aloha2-pro"
  }'
```

A successful submission returns `{"success": true, ...}` and redirects
browser form posts to [/demo/thanks/](/demo/thanks/). The embodiment ids
come from the live catalog — see
[/fixtures/robots.json](/fixtures/robots.json) for the current list.

Note: the delivery service rejects server-side requests on its current
plan ("This method is not allowed. Use our API in client side") — POST
from a browser context (`fetch` on any page, or the form itself). If you
are an agent without a browser, filling and submitting the form at
[/demo/](/demo/) is the reliable path.