# Sim triage

Screen at scale in simulation, verify the flagged slice on hardware, and recalibrate the sim with the ground truth that returns.

Your simulator can run a million episodes tonight; the lab cannot. The split
that works: simulation screens, physical testing verifies.
[`verify()`](/docs/primitives/verify/) is the interlock — it consumes the
initial conditions your simulator is least sure about, runs them on
instrumented hardware, and returns ground truth formatted to make the
simulator better. Each pass around the loop, the flagged set shrinks.

## The loop

### Screen at scale in your simulator

Run the big sweep in sim — the full perturbation space, more episodes than
you would ever pay hardware for. Rank scenarios by uncertainty: disagreement
across ensemble members, sensitivity to initial conditions, or plain
low-confidence predictions. The top slice is your candidate list.

### Export the flagged initial conditions

`verify()` accepts sampled initial states in PolaRiS, Isaac, and world-model
formats, or Roborama layout-replay specs — the same format every episode's
`replay_spec` uses, so scenarios from earlier physical runs are already
loadable.

### Verify them on hardware

Two arguments do the heavy lifting. `audit_sample=0.10` draws one episode in
ten at random from outside the flagged set. `return_ground_truth=True` ships
back mocap 6-DoF object poses and commanded-vs-executed trajectories for
every episode.

*Example: verify the flagged set*

**Python**

```python
import roborama  # reads ROBORAMA_API_KEY from the environment

policy = roborama.Policy.checkpoint(hf="acme/skill-v4", runtime="openpi")

# Initial conditions your simulator marked uncertain: sampled states from
# PolaRiS / Isaac / world-model exports, or Roborama layout-replay specs.
scenarios = roborama.scenarios.from_file("sim_flagged_p1.json")

report = roborama.verify(
    policy=policy,
    scenarios=scenarios,
    robots=["g1-edu-pro@fw2.3", "g1-edu-plus@fw2.3"],
    audit_sample=0.10,          # random episodes vs. the sim's blind spots
    return_ground_truth=True,   # mocap 6-DoF poses + commanded-vs-executed,
                                # formatted to recalibrate your simulator
)

print(report.sim_agreement)  # where reality matched sim predictions
print(report.disagreements)  # scenario ids where reality diverged
```

**TypeScript**

```typescript
import { readFileSync } from "node:fs";
import Roborama from "@roborama/sdk"; // reads ROBORAMA_API_KEY

const roborama = new Roborama();

// Initial conditions your simulator marked uncertain: sampled states from
// PolaRiS / Isaac / world-model exports, or Roborama layout-replay specs.
const scenarios = JSON.parse(readFileSync("sim_flagged_p1.json", "utf8"));

const report = await roborama.runs.create({
  kind: "verify",
  policy: { type: "checkpoint", hf: "acme/skill-v4", runtime: "openpi" },
  scenarios,
  robots: ["g1-edu-pro@fw2.3", "g1-edu-plus@fw2.3"],
  audit_sample: 0.1, // random episodes vs. the sim's blind spots
  return_ground_truth: true, // mocap 6-DoF poses + commanded-vs-executed
});

console.log(report.sim_agreement); // where reality matched sim predictions
console.log(report.disagreements); // scenario ids where reality diverged
```

**cURL**

```bash
# sim_flagged_p1.json: initial conditions your simulator marked uncertain
# (PolaRiS / Isaac / world-model exports, or Roborama layout-replay specs)
curl https://api.roborama.com/v1/runs \
  -H "Authorization: Bearer $ROBORAMA_API_KEY" \
  -H "Content-Type: application/json" \
  -d "$(jq -n --slurpfile s sim_flagged_p1.json '{
    kind: "verify",
    policy: { type: "checkpoint", hf: "acme/skill-v4", runtime: "openpi" },
    scenarios: $s[0],
    robots: ["g1-edu-pro@fw2.3", "g1-edu-plus@fw2.3"],
    audit_sample: 0.10,
    return_ground_truth: true
  }')"
```

**Agent (tool-use payload)**

```json
{
  "type": "tool_use",
  "name": "roborama_verify",
  "input": {
    "policy": { "type": "checkpoint", "hf": "acme/skill-v4",
                "runtime": "openpi" },
    "scenarios": {
      "format": "layout-replay",
      "source": "sim_flagged_p1.json",
      "count": 318
    },
    "robots": ["g1-edu-pro@fw2.3", "g1-edu-plus@fw2.3"],
    "audit_sample": 0.1,
    "return_ground_truth": true
  }
}
```

### Read the report

`report.sim_agreement` is the rate at which physical outcomes matched the
sim's predictions — canonically 0.918 (n=318, ci95 0.883–0.944). Respectable,
and not the point. `report.disagreements` lists the scenario ids where
reality diverged; those episodes are the payload, and each ships with full
ground truth.

### Recalibrate and go again

Feed the mocap poses and the commanded-vs-executed pairs into your
simulator's system identification, re-screen, and re-verify. The
[data contract](/docs/concepts/data-contract/) documents the calibration
export formats. When the flagged set stops shrinking, you have found your
simulator's honest edge.

> **Why the audit slice matters:** The flagged set tests the sim where it already knows it is weak. But the
> sim's confidence is exactly what you can't trust — a simulator that is wrong
> *and sure* flags nothing. The random `audit_sample` slice checks predictions
> the sim was confident about, so systematic blind spots surface instead of
> compounding quietly.

## What the loop costs

The canonical pass put 318 episodes on hardware — the flagged slice plus its
audit — to triage a screening sweep that would have been unpayable at
physical rates. That is the economics of the whole workflow: episodes in sim
are effectively free, so spend them everywhere; episodes on hardware carry
[two meters](/docs/guides/billing-and-budgets/), so spend them exactly where
the sim's answer can't be trusted. Quote the flagged set first like any other
job, and let the disagreement episodes — not the agreement rate — decide what
you fix next.