CI gates

Physical verification as a CI check — run a frozen suite on every release ref and fail the build on regression.

A policy that regresses on hardware should fail CI the same way a broken unit test does. roborama.gate() registers exactly that: every ref matching a pattern triggers a frozen benchmark suite on pinned hardware, and the check fails if the release drops past your tolerance or opens a failure mode the baseline didn't have.

import roborama  # reads ROBORAMA_API_KEY from the environment

# CI gate (GitHub Action / API): runs the suite on every matching ref
# and fails the check if the release regresses.
gate = roborama.gate(
    on="release/*",
    suite="rbr-manip-core@v3",
    robots=["g1-edu-pro@fw2.3"],
    fail_if={"success_rate_drop_pts": 2.0, "new_collision": True},
)

print(gate.id, gate.status)
# webhook events: run.completed, episode.failed, threshold.crossed,
#                 regression.detected, estop.triggered

What the gate checks

The suite is a frozen bundle — eval() under the hood. rbr-manip-core@v3 is 5 tasks × 300 episodes, so the comparison is apples to apples across releases. fail_if takes two conditions here:

• success_rate_drop_pts: 2.0 — fail if the suite success rate lands more than 2.0 points below the baseline, the suite run from the last release that passed.
• new_collision: true — fail if a collision cluster appears where the baseline had none. A rate can hold steady while the failure mix gets worse; this catches that.

Gate runs appear as ordinary runs with kind=eval, so everything else on this site — episodes, artifacts, reports — applies to them unchanged.

The workflow

Gates are declarative: registering the same gate again is a no-op, so the workflow can register on every push and then wait for the verdict.

name: physical-verification

on:
  push:
    branches:
      - "release/**"

jobs:
  gate:
    runs-on: ubuntu-latest
    timeout-minutes: 360
    steps:
      - name: Register the gate
        env:
          ROBORAMA_API_KEY: ${{ secrets.ROBORAMA_API_KEY }}
        run: |
          curl -sS --fail https://api.roborama.com/v1/gates \
            -H "Authorization: Bearer $ROBORAMA_API_KEY" \
            -H "Content-Type: application/json" \
            -d '{
              "on": "release/*",
              "suite": "rbr-manip-core@v3",
              "robots": ["g1-edu-pro@fw2.3"],
              "fail_if": {
                "success_rate_drop_pts": 2.0,
                "new_collision": true
              }
            }'

      - name: Wait for the verdict
        env:
          ROBORAMA_API_KEY: ${{ secrets.ROBORAMA_API_KEY }}
        run: |
          while true; do
            run=$(curl -sS "https://api.roborama.com/v1/runs?kind=eval&limit=1" \
              -H "Authorization: Bearer $ROBORAMA_API_KEY" | jq '.data[0]')
            status=$(echo "$run" | jq -r '.status')
            echo "suite run status: $status"
            case "$status" in
              completed)
                echo "$run" | jq '{result: .result, regressions: .regressions}'
                if [ "$(echo "$run" | jq '.regressions | length')" -eq 0 ]; then
                  exit 0
                fi
                exit 1
                ;;
              failed|stopped)
                exit 1
                ;;
              *)
                sleep 120
                ;;
            esac
          done

The polling step holds a runner while robots work through the suite. That is fine for small gates; a full rbr-manip-core@v3 pass takes hours of wall clock and can outlast a hosted runner's six-hour cap. For those, point the run.completed and regression.detected webhooks at a receiver that sets the commit status instead — nothing waits, and the verdict still lands on the pull request.

The five webhook events

• run.completed — any run reached completed. The payload carries the full result: n, rate, CI, clusters, both meters, cost. The generic "verdict is in" signal.
• episode.failed — one failed episode, as it happens, with its cluster label and a video URL. Subscribe to triage while the suite is still running.
• threshold.crossed — a threshold() contract met its target on the iterate tier; escalation to the verification tier is queued.
• regression.detected — a gate or matrix comparison found a cell below baseline. This is the one to wire to your release process.
• estop.triggered — an emergency stop fired in a cell running your policy. The run stops, the episode is marked estop, and the cell is inspected before resuming; partial results are kept.

A regression.detected delivery, verbatim:

{
  "id": "evt_5150",
  "type": "regression.detected",
  "created": "2026-09-14T08:03:55Z",
  "data": {
    "run_id": "run_8901",
    "vs": "run_8821",
    "robot": "g1-edu-pro@fw2.4",
    "environment": "kitchen-std@v1.2",
    "delta_pts": -10.5,
    "cluster": "grasp_slip"
  }
}

Behind that delta: kitchen-std@v1.2 on fw2.4 came in at 0.787 (n=240, ci95 0.731–0.835) against 0.892 (n=240, ci95 0.846–0.925) on fw2.3 in run_8821. A wrist controller change opened a grasp_slip cluster — the webhook names the cluster so triage starts at the right episodes, not at a diff of the firmware release notes.

Watch it live

Gate runs stream like any other run: an episode ticker plus WebRTC streams for the cell doing the work.

import roborama  # reads ROBORAMA_API_KEY from the environment

run = roborama.runs.get("run_8842")

for event in run.watch():   # live: episode ticker + WebRTC stream URLs
    print(event.episode, event.status)

stream = roborama.streams.get("cell-g1-04")
print(stream)
# {webrtc: "wss://streams.roborama.com/cells/cell-g1-04/webrtc",
#  mjpeg: "https://streams.roborama.com/cells/cell-g1-04/mjpeg",
#  viewer_token: "vt_7Kq2mHentXw4"}