eval()

Frozen benchmark suites with attested, citable reports — comparable across every team that ran the same suite revision.

roborama.run() answers a question about your task. roborama.eval() answers a question other people can check. It runs your policy against a frozen benchmark suite and returns a report with an attestation anyone can verify — no Roborama account required.

The call

import roborama  # reads ROBORAMA_API_KEY from the environment

policy = roborama.Policy.checkpoint(hf="acme/skill-v4", runtime="openpi")

report = roborama.eval(
    policy=policy,
    suite="rbr-manip-core@v3",  # versioned suite (5 tasks × 300 eps)
    robots=["g1-edu-pro@fw2.3"],
    publish="private",           # "private" | "leaderboard" (opt-in only)
)

print(report.citation)
# Roborama Manip-Core v3, run 2026-09-04, attestation rbr:att:9f3c2ab7
print(report.attestation_url)   # verifiable benchmark attestation

Frozen suites

A suite is a frozen bundle: tasks, environments, perturbation schedules, and scoring, pinned under one revision. rbr-manip-core@v3 is five tasks × 300 episodes — pick_place@v1, load_dishwasher@v2, shelf_restock@v1, fold_towel@v2, bin_pick@v1 — across kitchen-std@v1.2, warehouse-std@v2.0, and cell-a@v1.0, with perturbation schedules pinned at suite-core@v3.0 and binary success scored with Wilson 95% intervals. The bundle froze on 2026-03-14 and nothing in it moves: a v3 score next year means what a v3 score means today. When the suite needs to change, that is a new revision, and revisions are never compared to each other.

A sample report

The canonical g1-edu-pro@fw2.3 report on rbr-manip-core@v3:

The per-task rows are the point. An overall 0.859 (n=1500, ci95 0.841–0.876) looks healthy; the table shows bin_pick@v1 at 0.777 (n=300, ci95 0.726–0.82) dragging it, which is where your next training run should look.

Citation and attestation

report.citation returns a string built for a paper or a launch post:

Roborama Manip-Core v3, run 2026-09-04, attestation rbr:att:9f3c2ab7

The attestation id rbr:att:9f3c2ab7 resolves at report.attestation_url — https://api.roborama.com/v1/attestations/rbr:att:9f3c2ab7 — where anyone can check the n, the rates, the intervals, and every pin behind them. A claim that carries its own audit trail.

Publishing

Results are private. publish="private" is the default posture, and publish="leaderboard" is strictly opt-in: it places the attested report on the public leaderboard for that suite revision, where it is comparable with every other opted-in result on the same revision. Nothing is published by accident; a key without the right scope gets publish_forbidden back.

Where next

• run() — your own task, your own perturbation schedule.
• matrix() — coverage across embodiments × environments.
• CI gates — run the suite on release/* and fail the build on a drop.