verify()

Run the scenarios your simulator flagged on instrumented hardware — get back agreement, disagreements, and ground truth to recalibrate.

Simulation screens; physical testing verifies. roborama.verify() is the interlock between the two: it consumes the initial conditions your simulator is least sure about, runs them on instrumented hardware, and returns where reality agreed with the sim — and, more usefully, where it didn't, with the ground truth to fix it.

The call

import roborama  # reads ROBORAMA_API_KEY from the environment

policy = roborama.Policy.checkpoint(hf="acme/skill-v4", runtime="openpi")

# Initial conditions your simulator marked uncertain: sampled states from
# PolaRiS / Isaac / world-model exports, or Roborama layout-replay specs.
scenarios = roborama.scenarios.from_file("sim_flagged_p1.json")

report = roborama.verify(
    policy=policy,
    scenarios=scenarios,
    robots=["g1-edu-pro@fw2.3", "g1-edu-plus@fw2.3"],
    audit_sample=0.10,          # random episodes vs. the sim's blind spots
    return_ground_truth=True,   # mocap 6-DoF poses + commanded-vs-executed,
                                # formatted to recalibrate your simulator
)

print(report.sim_agreement)  # where reality matched sim predictions
print(report.disagreements)  # scenario ids where reality diverged

Scenarios your sim flagged

scenarios accepts sampled initial states in PolaRiS, Isaac, and world-model formats, or Roborama layout-replay specs — scene layouts as JSON or USD. The usual source is something like sim_report.flagged(percentile=1): the slice of scenarios where your simulator's own confidence is worst. That targeting is the economics of the primitive — robot-hours go to the episodes where the sim is least trustworthy instead of being spread uniformly over scenes the sim already handles well. A malformed scene file fails fast with scenario_format_invalid, before anything is scheduled.

Auditing the blind spots

A flagged set has a failure mode of its own: it only contains what the simulator knows it doesn't know. audit_sample=0.10 spends a tenth of the episode budget on randomly drawn scenarios instead — a control arm against the sim's blind spots. If the audit arm disagrees with the sim more than the flagged arm does, your simulator's uncertainty estimates are themselves miscalibrated, and that is worth knowing before you trust its next screen.

Ground truth that recalibrates

With return_ground_truth=True, every episode ships motion-capture 6-DoF object poses and commanded-versus-executed joint traces — in MCAP channels /gt/object_poses, /joint_targets_commanded, and /joint_states_measured — formatted to load into a simulator calibration pipeline. You don't just learn that reality diverged; you get the trajectories to make your simulator diverge less next time.

Agreement, and the payload

report.sim_agreement is the headline: 0.918 (n=318, ci95 0.883–0.944) of physical outcomes matched the sim's predictions on this set. report.disagreements is the payload: the scenario ids where reality diverged, each shipping with full ground truth. Read them together — agreement tells you how far to trust the simulator, disagreements tell you exactly which scenarios to re-examine and what actually happened in them. This is the calibration loop that makes sim vendors customers.

Where next

• Sim triage — the workflow: flag, verify, recalibrate, repeat.
• Data contract — what comes back, in which formats, and who owns it.
• run() — the underlying primitive, field by field.