# eval()

Frozen benchmark suites with attested, citable reports — comparable across every team that ran the same suite revision.

`roborama.run()` answers a question about your task. `roborama.eval()` answers
a question other people can check. It runs your policy against a frozen
benchmark suite and returns a report with an attestation anyone can verify —
no Roborama account required.

## The call

*Example: roborama.eval()*

**Python**

```python
import roborama  # reads ROBORAMA_API_KEY from the environment

policy = roborama.Policy.checkpoint(hf="acme/skill-v4", runtime="openpi")

report = roborama.eval(
    policy=policy,
    suite="rbr-manip-core@v3",  # versioned suite (5 tasks × 300 eps)
    robots=["g1-edu-pro@fw2.3"],
    publish="private",           # "private" | "leaderboard" (opt-in only)
)

print(report.citation)
# Roborama Manip-Core v3, run 2026-09-04, attestation rbr:att:9f3c2ab7
print(report.attestation_url)   # verifiable benchmark attestation
```

**TypeScript**

```typescript
import Roborama from "@roborama/sdk"; // reads ROBORAMA_API_KEY

const roborama = new Roborama();

const report = await roborama.runs.create({
  kind: "eval",
  policy: { type: "checkpoint", hf: "acme/skill-v4", runtime: "openpi" },
  suite: "rbr-manip-core@v3", // versioned suite (5 tasks × 300 eps)
  robots: ["g1-edu-pro@fw2.3"],
  publish: "private", // "private" | "leaderboard" (opt-in only)
});

console.log(report.citation);
// Roborama Manip-Core v3, run 2026-09-04, attestation rbr:att:9f3c2ab7
console.log(report.attestation_url);
```

**cURL**

```bash
curl https://api.roborama.com/v1/runs \
  -H "Authorization: Bearer $ROBORAMA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "kind": "eval",
    "policy": { "type": "checkpoint", "hf": "acme/skill-v4",
                "runtime": "openpi" },
    "suite": "rbr-manip-core@v3",
    "robots": ["g1-edu-pro@fw2.3"],
    "publish": "private"
  }'
```

**Agent (tool-use payload)**

```json
{
  "type": "tool_use",
  "name": "roborama_eval",
  "input": {
    "policy": { "type": "checkpoint", "hf": "acme/skill-v4",
                "runtime": "openpi" },
    "suite": "rbr-manip-core@v3",
    "robots": ["g1-edu-pro@fw2.3"],
    "publish": "private"
  }
}
```

## Frozen suites

A suite is a frozen bundle: tasks, environments, perturbation schedules, and
scoring, pinned under one revision. `rbr-manip-core@v3` is five tasks × 300
episodes — `pick_place@v1`, `load_dishwasher@v2`, `shelf_restock@v1`,
`fold_towel@v2`, `bin_pick@v1` — across `kitchen-std@v1.2`,
`warehouse-std@v2.0`, and `cell-a@v1.0`, with perturbation schedules pinned at
`suite-core@v3.0` and binary success scored with Wilson 95% intervals. The
bundle froze on 2026-03-14 and nothing in it moves: a v3 score next year means
what a v3 score means today. When the suite needs to change, that is a new
revision, and revisions are never compared to each other.

> **Comparable by construction:** Two teams that run the same suite revision ran the same tasks, the same
> scenes, the same perturbation schedules, and the same scoring. Their numbers
> sit on one axis. That is the property that makes independent citation
> possible.

## A sample report

The canonical `g1-edu-pro@fw2.3` report on `rbr-manip-core@v3`:

| Task | n | Success rate | ci95 |
| --- | --- | --- | --- |
| `pick_place@v1` | 300 | 0.903 | 0.865–0.932 |
| `load_dishwasher@v2` | 300 | 0.873 | 0.831–0.906 |
| `shelf_restock@v1` | 300 | 0.813 | 0.765–0.853 |
| `fold_towel@v2` | 300 | 0.93 | 0.895–0.954 |
| `bin_pick@v1` | 300 | 0.777 | 0.726–0.82 |
| Overall | 1500 | 0.859 | 0.841–0.876 |

The per-task rows are the point. An overall 0.859 (n=1500, ci95 0.841–0.876)
looks healthy; the table shows `bin_pick@v1` at 0.777 (n=300, ci95 0.726–0.82)
dragging it, which is where your next training run should look.

## Citation and attestation

`report.citation` returns a string built for a paper or a launch post:

```text
Roborama Manip-Core v3, run 2026-09-04, attestation rbr:att:9f3c2ab7
```

The attestation id `rbr:att:9f3c2ab7` resolves at `report.attestation_url` —
`https://api.roborama.com/v1/attestations/rbr:att:9f3c2ab7` — where anyone can
check the n, the rates, the intervals, and every pin behind them. A claim that
carries its own audit trail.

## Publishing

Results are private. `publish="private"` is the default posture, and
`publish="leaderboard"` is strictly opt-in: it places the attested report on
the public leaderboard for that suite revision, where it is comparable with
every other opted-in result on the same revision. Nothing is published by
accident; a key without the right scope gets `publish_forbidden` back.

## Where next

- [run()](/docs/primitives/run/) — your own task, your own perturbation schedule.
- [matrix()](/docs/primitives/matrix/) — coverage across embodiments × environments.
- [CI gates](/docs/guides/ci-gates/) — run the suite on `release/*` and fail the build on a drop.