Get a demoSign up
Documentation menu

Getting started

Concepts

Bring your policy

Primitives

API reference

SDKs

Guides

Reference

Evaluate a LeRobot checkpoint from Hugging Face

An ACT policy trained with LeRobot, published to the Hub as a release repo, and measured on real bimanual hardware — repo to report for $860.

Hugging Face

LeRobot is a Hugging Face project, so a trained LeRobot policy's natural home is a Hub repo — and a repo id is all Policy.checkpoint needs. This walkthrough follows a team that trained an ACT policy with LeRobot, published it as acme/act-so101-pick-v2, and wants a number they can defend in a release review: a success rate on real hardware, with an n, a Wilson interval, and named failure modes.

  1. Know what the repo carries

    The training run left a pretrained_model/ directory; publishing it is one command:

    huggingface-cli upload acme/act-so101-pick-v2 ./pretrained_model

    The repo now holds the complete policy, not just weights:

    acme/act-so101-pick-v2
├── config.json                 # architecture + hyperparameters
├── model.safetensors           # the weights
├── train_config.json           # full training provenance
├── policy_preprocessor.json    # observation normalization
└── policy_postprocessor.json   # action de-normalization

    The processor files are the part teams forget: they carry the normalization and pre/post-processing the policy was trained with. Two checkpoints with identical weights and different preprocessing are different policies — which is why a weights hash alone does not identify what you evaluated, and why the lerobot runtime loads the repo as a unit.

    There is no revision field on Policy.checkpoint; the version lives in the repo id. -v2 is a release repo — when v3 ships, it gets its own, and acme/act-so101-pick-v2 stays exactly what v2's report cites. The hf: field reads repos that are readable without authentication; private or gated weights take the container route instead.

  2. Pick hardware the policy can actually drive

    From the catalogue: the bimanual verification rig aloha2-pro@fw1.1 ($74/robot-hour), the bare bench cell cell-a@v1.0 ($12/env-hour) — enough scene for pick_place@v1 — and the two contracts the policy declares: joint_delta_50hz actions, the droid-3cam observation set.

    Compatibility is declared, not assumed: admission validates both declarations against the pinned robot and scene, and a mismatch is rejected as contract_validation_failed before any motor moves — a rejected submission costs nothing.

  3. Dry-run the loop locally

    Set ROBORAMA_MOCK=1 and run the script from step 5 unchanged: no API key, no hardware — the full quote → submit → watch → result loop answered from packaged fixtures. The mock-mode snippet lives on Bring your policy. After the local pass come the other two layers, in the order you hit them: admission (free, as above), then checkpoint fetch at cell startup.

  4. Quote the job

    A fixed episodes=300 buys an interval about ±4 points wide at the rate this policy lands — a deliberate sizing, not a default. At $74 per robot-hour, $12 per environment-hour, and the task's 2.0-minute episode estimate, 300 episodes put 10.0 hours on each meter:

    {
  "object": "quote",
  "robot": "aloha2-pro",
  "environment": "cell-a",
  "episodes": 300,
  "priority": "standard",
  "robot_hours": 10.0,
  "env_hours": 10.0,
  "usd": 860,
  "queue_eta": "2h",
  "breakdown": {
    "minutes_per_episode_est": 2.0,
    "robot_usd_per_hour": 74,
    "env_usd_per_hour": 12,
    "robot_usd": 740,
    "env_usd": 120
  },
  "expires": "2026-09-18T00:00:00Z"
}

    Quotes are free, and nothing bills until a motor moves — see Billing & budgets.

  5. Submit and follow
    the whole loop
    import roborama  # reads ROBORAMA_API_KEY from the environment

# 1. Quote — free; both meters visible before anything books
quote = roborama.quote(
    robot="aloha2-pro",
    environment="cell-a",
    episodes=300,
)
print(quote)
# {robot_hours: 10.0, env_hours: 10.0, usd: 860, queue_eta: "2h"}

# 2. The policy — a LeRobot repo on the Hub; version in the repo id
policy = roborama.Policy.checkpoint(
    hf="acme/act-so101-pick-v2",
    runtime="lerobot",
    inference={"action_horizon": 100, "chunk_size": 100, "temp": 0.0},
    action_space="joint_delta_50hz",
    observation_contract="droid-3cam",
)

# 3. Submit — contracts validated before any motor moves
run = roborama.run(
    robot="aloha2-pro@fw1.1",
    environment="cell-a@v1.0",
    policy=policy,
    task="pick_place@v1",
    episodes=300,
    max_budget_usd=1000,
)

# 4. Follow — status transitions, then one tick per episode
for event in run.watch():
    print(event.episode, event.status)

# 5. The verdict
print(run.result())
# n=300  rate=0.843  ci95=(0.797, 0.881)
# failure_clusters: {grasp_slip: 31, perception_miss: 16}
# robot_hours=10.0  environment_hours=10.0  cost_usd=860

    max_budget_usd=1000 is the hard stop over the $860 estimate: if a stuck reset ever burned through the headroom, the run would halt with budget_exceeded and keep the completed episodes with honest statistics.

    The watch loop prints one event per line — status transitions, then an episode tick per episode. (run.watch() rides the same SSE stream the REST API exposes.)

    status    queued       queue_eta=2h
status    running      cell=cell-b-01
episode   1/300        outcome=success   duration_s=118
episode   2/300        outcome=success   duration_s=112
episode   3/300        outcome=failure   cluster=grasp_slip
[296 more episode events]
episode   300/300      outcome=success   duration_s=124
status    completed
  6. Read the result, keep the report

    run_9284 came back at 0.843 — with the interval and the failure mix that make the number usable:

    {
  "n": 300,
  "success_rate": 0.843,
  "ci95": [0.797, 0.881],
  "interval_method": "wilson",
  "failure_clusters": {
    "grasp_slip": 31,
    "perception_miss": 16
  },
  "robot_hours": 10.0,
  "environment_hours": 10.0,
  "cost_usd": 860
}

    Two-thirds of the failures are grasp_slip — episode 3 above was one, and run.episodes links every clustered episode to its video and MCAP telemetry. run.report(format="pdf") renders the verification report: the rate, the interval, and every resolved pin — robot firmware, scene revision, task version, the inference dict — citable in the review.

    The fetched checkpoint is cached only to execute the run and is deleted when the run's data.retention window closes — or immediately on roborama.data.purge(run_id), receipted. Making the repo private later does not delete an already-fetched copy; purge does (data contract).

  7. When v3 ships, compare — don't re-argue

    Publish acme/act-so101-pick-v3 as its own release repo and keep the declared test configuration identical — same aloha2-pro@fw1.1 pin, same cell-a@v1.0 revision, same task and contracts — so the two reports differ in exactly one input: the repo id. Better still, roborama.compare() runs both checkpoints in paired episodes with the same initial conditions and separates them in about half the episodes two independent runs would need.

Where next