Sim triage

Screen at scale in simulation, verify the flagged slice on hardware, and recalibrate the sim with the ground truth that returns.

Your simulator can run a million episodes tonight; the lab cannot. The split that works: simulation screens, physical testing verifies. verify() is the interlock — it consumes the initial conditions your simulator is least sure about, runs them on instrumented hardware, and returns ground truth formatted to make the simulator better. Each pass around the loop, the flagged set shrinks.

The loop

1. Screen at scale in your simulator

   Run the big sweep in sim — the full perturbation space, more episodes than you would ever pay hardware for. Rank scenarios by uncertainty: disagreement across ensemble members, sensitivity to initial conditions, or plain low-confidence predictions. The top slice is your candidate list.

2. Export the flagged initial conditions

   verify() accepts sampled initial states in PolaRiS, Isaac, and world-model formats, or Roborama layout-replay specs — the same format every episode's replay_spec uses, so scenarios from earlier physical runs are already loadable.

3. Verify them on hardware

   Two arguments do the heavy lifting. audit_sample=0.10 draws one episode in ten at random from outside the flagged set. return_ground_truth=True ships back mocap 6-DoF object poses and commanded-vs-executed trajectories for every episode.

   verify the flagged setPythonTypeScriptcURLAgentcopycopy .md

   import roborama  # reads ROBORAMA_API_KEY from the environment
   
   policy = roborama.Policy.checkpoint(hf="acme/skill-v4", runtime="openpi")
   
   # Initial conditions your simulator marked uncertain: sampled states from
   # PolaRiS / Isaac / world-model exports, or Roborama layout-replay specs.
   scenarios = roborama.scenarios.from_file("sim_flagged_p1.json")
   
   report = roborama.verify(
       policy=policy,
       scenarios=scenarios,
       robots=["g1-edu-pro@fw2.3", "g1-edu-plus@fw2.3"],
       audit_sample=0.10,          # random episodes vs. the sim's blind spots
       return_ground_truth=True,   # mocap 6-DoF poses + commanded-vs-executed,
                                   # formatted to recalibrate your simulator
   )
   
   print(report.sim_agreement)  # where reality matched sim predictions
   print(report.disagreements)  # scenario ids where reality diverged

4. Read the report

   report.sim_agreement is the rate at which physical outcomes matched the sim's predictions — canonically 0.918 (n=318, ci95 0.883–0.944). Respectable, and not the point. report.disagreements lists the scenario ids where reality diverged; those episodes are the payload, and each ships with full ground truth.

5. Recalibrate and go again

   Feed the mocap poses and the commanded-vs-executed pairs into your simulator's system identification, re-screen, and re-verify. The data contract documents the calibration export formats. When the flagged set stops shrinking, you have found your simulator's honest edge.

What the loop costs

The canonical pass put 318 episodes on hardware — the flagged slice plus its audit — to triage a screening sweep that would have been unpayable at physical rates. That is the economics of the whole workflow: episodes in sim are effectively free, so spend them everywhere; episodes on hardware carry two meters, so spend them exactly where the sim's answer can't be trusted. Quote the flagged set first like any other job, and let the disagreement episodes — not the agreement rate — decide what you fix next.