# Evaluate a LeRobot checkpoint from Hugging Face

An ACT policy trained with LeRobot, published to the Hub as a release repo, and measured on real bimanual hardware — repo to report for $860.

*Provider workflow: Hugging Face.*

LeRobot is a Hugging Face project, so a trained LeRobot policy's natural
home is a Hub repo — and a repo id is all `Policy.checkpoint` needs. This
walkthrough follows a team that trained an ACT policy with LeRobot,
published it as `acme/act-so101-pick-v2`, and wants a number they can
defend in a release review: a success rate on real hardware, with an n,
a Wilson interval, and named failure modes.

### Know what the repo carries

The training run left a `pretrained_model/` directory; publishing it is
one command:

```bash
huggingface-cli upload acme/act-so101-pick-v2 ./pretrained_model
```

The repo now holds the complete policy, not just weights:

```text
acme/act-so101-pick-v2
├── config.json                 # architecture + hyperparameters
├── model.safetensors           # the weights
├── train_config.json           # full training provenance
├── policy_preprocessor.json    # observation normalization
└── policy_postprocessor.json   # action de-normalization
```

The processor files are the part teams forget: they carry the
normalization and pre/post-processing the policy was trained with. Two
checkpoints with identical weights and different preprocessing are
different policies — which is why a weights hash alone does not identify
what you evaluated, and why the `lerobot` runtime loads the repo as a
unit.

There is no `revision` field on `Policy.checkpoint`; the version lives
in the repo id. `-v2` is a release repo — when v3 ships, it gets its
own, and `acme/act-so101-pick-v2` stays exactly what v2's report cites.
The `hf:` field reads repos that are readable without authentication;
private or gated weights take the
[container route](/docs/guides/package-your-policy/) instead.

### Pick hardware the policy can actually drive

From the catalogue: the bimanual verification rig `aloha2-pro@fw1.1`
($74/robot-hour), the bare bench cell `cell-a@v1.0` ($12/env-hour) —
enough scene for `pick_place@v1` — and the two contracts the policy
declares: `joint_delta_50hz` actions, the `droid-3cam` observation set.

Compatibility is declared, not assumed: admission validates both
declarations against the pinned robot and scene, and a mismatch is
rejected as `contract_validation_failed` before any motor moves — a
rejected submission costs nothing.

### Dry-run the loop locally

Set `ROBORAMA_MOCK=1` and run the script from step 5 unchanged: no API
key, no hardware — the full quote → submit → watch → result loop
answered from packaged fixtures. The mock-mode snippet lives on
[Bring your policy](/docs/guides/bring-your-policy/). After the local
pass come the other two layers, in the order you hit them: admission
(free, as above), then checkpoint fetch at cell startup.

### Quote the job

A fixed `episodes=300` buys an interval about ±4 points wide at the
rate this policy lands — a deliberate sizing, not a default. At $74 per
robot-hour, $12 per environment-hour, and the task's 2.0-minute episode
estimate, 300 episodes put 10.0 hours on each meter:

```json
{
  "object": "quote",
  "robot": "aloha2-pro",
  "environment": "cell-a",
  "episodes": 300,
  "priority": "standard",
  "robot_hours": 10.0,
  "env_hours": 10.0,
  "usd": 860,
  "queue_eta": "2h",
  "breakdown": {
    "minutes_per_episode_est": 2.0,
    "robot_usd_per_hour": 74,
    "env_usd_per_hour": 12,
    "robot_usd": 740,
    "env_usd": 120
  },
  "expires": "2026-09-18T00:00:00Z"
}
```

> **Fixture output:** The ids and results on this page — the $860 quote, `run_9284` and its
> statistics — are documentation fixtures. The numbers are internally
> consistent (meter math, Wilson intervals), but they are not live API
> responses.

Quotes are free, and nothing bills until a motor moves — see
[Billing & budgets](/docs/guides/billing-and-budgets/).

### Submit and follow

*Example: the whole loop*

**Python**

```python
import roborama  # reads ROBORAMA_API_KEY from the environment

# 1. Quote — free; both meters visible before anything books
quote = roborama.quote(
    robot="aloha2-pro",
    environment="cell-a",
    episodes=300,
)
print(quote)
# {robot_hours: 10.0, env_hours: 10.0, usd: 860, queue_eta: "2h"}

# 2. The policy — a LeRobot repo on the Hub; version in the repo id
policy = roborama.Policy.checkpoint(
    hf="acme/act-so101-pick-v2",
    runtime="lerobot",
    inference={"action_horizon": 100, "chunk_size": 100, "temp": 0.0},
    action_space="joint_delta_50hz",
    observation_contract="droid-3cam",
)

# 3. Submit — contracts validated before any motor moves
run = roborama.run(
    robot="aloha2-pro@fw1.1",
    environment="cell-a@v1.0",
    policy=policy,
    task="pick_place@v1",
    episodes=300,
    max_budget_usd=1000,
)

# 4. Follow — status transitions, then one tick per episode
for event in run.watch():
    print(event.episode, event.status)

# 5. The verdict
print(run.result())
# n=300  rate=0.843  ci95=(0.797, 0.881)
# failure_clusters: {grasp_slip: 31, perception_miss: 16}
# robot_hours=10.0  environment_hours=10.0  cost_usd=860
```

**TypeScript**

```typescript
import Roborama from "@roborama/sdk"; // reads ROBORAMA_API_KEY

const roborama = new Roborama();

// 1. Quote — free; both meters visible before anything books
const quote = await roborama.quote({
  robot: "aloha2-pro",
  environment: "cell-a",
  episodes: 300,
});
console.log(quote);
// { robot_hours: 10.0, env_hours: 10.0, usd: 860, queue_eta: "2h" }

// 2. The policy — a LeRobot repo on the Hub; version in the repo id
const policy = {
  type: "checkpoint",
  hf: "acme/act-so101-pick-v2",
  runtime: "lerobot",
  inference: { action_horizon: 100, chunk_size: 100, temp: 0.0 },
  action_space: "joint_delta_50hz",
  observation_contract: "droid-3cam",
} as const;

// 3. Submit — contracts validated before any motor moves
const run = await roborama.runs.create({
  robot: "aloha2-pro@fw1.1",
  environment: "cell-a@v1.0",
  policy,
  task: "pick_place@v1",
  episodes: 300,
  max_budget_usd: 1000,
});

// 4. Follow — status transitions, then one tick per episode
for await (const event of run.watch()) {
  console.log(event.episode, event.status);
}

// 5. The verdict
console.log(await run.result());
// n=300  rate=0.843  ci95=(0.797, 0.881)
// failure_clusters: { grasp_slip: 31, perception_miss: 16 }
// robot_hours=10.0  environment_hours=10.0  cost_usd=860
```

**cURL**

```bash
# 1. Quote — free; both meters visible before anything books
curl -sS https://api.roborama.com/v1/quote \
  -H "Authorization: Bearer $ROBORAMA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "robot": "aloha2-pro",
    "environment": "cell-a",
    "episodes": 300
  }'
# {"robot_hours": 10.0, "env_hours": 10.0, "usd": 860}

# 2 + 3. Submit — contracts validated before any motor moves
curl -sS https://api.roborama.com/v1/runs \
  -H "Authorization: Bearer $ROBORAMA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "kind": "run",
    "robot": "aloha2-pro@fw1.1",
    "environment": "cell-a@v1.0",
    "policy": {
      "type": "checkpoint",
      "hf": "acme/act-so101-pick-v2",
      "runtime": "lerobot",
      "inference": {
        "action_horizon": 100,
        "chunk_size": 100,
        "temp": 0.0
      },
      "action_space": "joint_delta_50hz",
      "observation_contract": "droid-3cam"
    },
    "task": "pick_place@v1",
    "episodes": 300,
    "max_budget_usd": 1000
  }'

# 4. Follow the run live (server-sent events)
curl -N https://api.roborama.com/v1/runs/run_9284/events \
  -H "Authorization: Bearer $ROBORAMA_API_KEY"

# 5. The verdict
curl -sS https://api.roborama.com/v1/runs/run_9284 \
  -H "Authorization: Bearer $ROBORAMA_API_KEY"
# n=300 rate=0.843 ci95=(0.797, 0.881) cost_usd=860
```

**Agent (tool-use payload)**

```json
[
  {
    "type": "tool_use",
    "name": "roborama_quote",
    "input": {
      "robot": "aloha2-pro",
      "environment": "cell-a",
      "episodes": 300
    }
  },
  {
    "type": "tool_use",
    "name": "roborama_run",
    "input": {
      "robot": "aloha2-pro@fw1.1",
      "environment": "cell-a@v1.0",
      "policy": {
        "type": "checkpoint",
        "hf": "acme/act-so101-pick-v2",
        "runtime": "lerobot",
        "inference": {
          "action_horizon": 100,
          "chunk_size": 100,
          "temp": 0.0
        },
        "action_space": "joint_delta_50hz",
        "observation_contract": "droid-3cam"
      },
      "task": "pick_place@v1",
      "episodes": 300,
      "max_budget_usd": 1000
    }
  },
  {
    "type": "tool_use",
    "name": "roborama_watch_run",
    "input": { "run_id": "run_9284" }
  }
]
```

`max_budget_usd=1000` is the hard stop over the $860 estimate: if a
stuck reset ever burned through the headroom, the run would halt with
`budget_exceeded` and keep the completed episodes with honest
statistics.

The watch loop prints one event per line — status transitions, then an
episode tick per episode. (`run.watch()` rides the same SSE stream the
[REST API](/docs/sdks/rest/) exposes.)

```text
status    queued       queue_eta=2h
status    running      cell=cell-b-01
episode   1/300        outcome=success   duration_s=118
episode   2/300        outcome=success   duration_s=112
episode   3/300        outcome=failure   cluster=grasp_slip
[296 more episode events]
episode   300/300      outcome=success   duration_s=124
status    completed
```

### Read the result, keep the report

`run_9284` came back at 0.843 — with the interval and the failure mix
that make the number usable:

```json
{
  "n": 300,
  "success_rate": 0.843,
  "ci95": [0.797, 0.881],
  "interval_method": "wilson",
  "failure_clusters": {
    "grasp_slip": 31,
    "perception_miss": 16
  },
  "robot_hours": 10.0,
  "environment_hours": 10.0,
  "cost_usd": 860
}
```

Two-thirds of the failures are `grasp_slip` — episode 3 above was one,
and `run.episodes` links every clustered episode to its video and MCAP
telemetry. `run.report(format="pdf")` renders the verification report:
the rate, the interval, and every resolved pin — robot firmware, scene
revision, task version, the `inference` dict — citable in the review.

The fetched checkpoint is cached only to execute the run and is deleted
when the run's `data.retention` window closes — or immediately on
`roborama.data.purge(run_id)`, receipted. Making the repo private later
does not delete an already-fetched copy; purge does
([data contract](/docs/concepts/data-contract/)).

### When v3 ships, compare — don't re-argue

Publish `acme/act-so101-pick-v3` as its own release repo and keep the
declared test configuration identical — same `aloha2-pro@fw1.1` pin,
same `cell-a@v1.0` revision, same task and contracts — so the two
reports differ in exactly one input: the repo id. Better still,
[`roborama.compare()`](/docs/primitives/compare/) runs both checkpoints
in paired episodes with the same initial conditions and separates them
in about half the episodes two independent runs would need.

## Where next

- [Import from Hugging Face](/docs/guides/import-from-hugging-face/) — the checkpoint path in full: runtimes, cache, and the private-repo boundary.
- [Bring your policy](/docs/guides/bring-your-policy/) — all three packagings, plus the mock-mode dry run.
- [compare()](/docs/primitives/compare/) — paired A/B for v3 against v2.
- [Billing & budgets](/docs/guides/billing-and-budgets/) — quotes, `max_budget_usd`, and both meters.