# CI gates

Physical verification as a CI check — run a frozen suite on every release ref and fail the build on regression.

A policy that regresses on hardware should fail CI the same way a broken unit
test does. `roborama.gate()` registers exactly that: every ref matching a
pattern triggers a frozen benchmark suite on pinned hardware, and the check
fails if the release drops past your tolerance or opens a failure mode the
baseline didn't have.

*Example: the gate*

**Python**

```python
import roborama  # reads ROBORAMA_API_KEY from the environment

# CI gate (GitHub Action / API): runs the suite on every matching ref
# and fails the check if the release regresses.
gate = roborama.gate(
    on="release/*",
    suite="rbr-manip-core@v3",
    robots=["g1-edu-pro@fw2.3"],
    fail_if={"success_rate_drop_pts": 2.0, "new_collision": True},
)

print(gate.id, gate.status)
# webhook events: run.completed, episode.failed, threshold.crossed,
#                 regression.detected, estop.triggered
```

**TypeScript**

```typescript
import Roborama from "@roborama/sdk"; // reads ROBORAMA_API_KEY

const roborama = new Roborama();

// CI gate (GitHub Action / API): runs the suite on every matching ref
// and fails the check if the release regresses.
const gate = await roborama.gates.create({
  on: "release/*",
  suite: "rbr-manip-core@v3",
  robots: ["g1-edu-pro@fw2.3"],
  fail_if: { success_rate_drop_pts: 2.0, new_collision: true },
});

console.log(gate.id, gate.status);
// webhook events: run.completed, episode.failed, threshold.crossed,
//                 regression.detected, estop.triggered
```

**cURL**

```bash
curl https://api.roborama.com/v1/gates \
  -H "Authorization: Bearer $ROBORAMA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "on": "release/*",
    "suite": "rbr-manip-core@v3",
    "robots": ["g1-edu-pro@fw2.3"],
    "fail_if": { "success_rate_drop_pts": 2.0, "new_collision": true }
  }'
```

**Agent (tool-use payload)**

```json
{
  "type": "tool_use",
  "name": "roborama_create_gate",
  "input": {
    "on": "release/*",
    "suite": "rbr-manip-core@v3",
    "robots": ["g1-edu-pro@fw2.3"],
    "fail_if": { "success_rate_drop_pts": 2.0, "new_collision": true }
  }
}
```

## What the gate checks

The suite is a frozen bundle — [`eval()`](/docs/primitives/eval/) under the
hood. `rbr-manip-core@v3` is 5 tasks × 300 episodes, so the comparison is
apples to apples across releases. `fail_if` takes two conditions here:

- `success_rate_drop_pts: 2.0` — fail if the suite success rate lands more
  than 2.0 points below the baseline, the suite run from the last release
  that passed.
- `new_collision: true` — fail if a collision cluster appears where the
  baseline had none. A rate can hold steady while the failure mix gets worse;
  this catches that.

Gate runs appear as ordinary runs with `kind=eval`, so everything else on this
site — episodes, artifacts, reports — applies to them unchanged.

## The workflow

Gates are declarative: registering the same gate again is a no-op, so the
workflow can register on every push and then wait for the verdict.

```yaml
name: physical-verification

on:
  push:
    branches:
      - "release/**"

jobs:
  gate:
    runs-on: ubuntu-latest
    timeout-minutes: 360
    steps:
      - name: Register the gate
        env:
          ROBORAMA_API_KEY: ${{ secrets.ROBORAMA_API_KEY }}
        run: |
          curl -sS --fail https://api.roborama.com/v1/gates \
            -H "Authorization: Bearer $ROBORAMA_API_KEY" \
            -H "Content-Type: application/json" \
            -d '{
              "on": "release/*",
              "suite": "rbr-manip-core@v3",
              "robots": ["g1-edu-pro@fw2.3"],
              "fail_if": {
                "success_rate_drop_pts": 2.0,
                "new_collision": true
              }
            }'

      - name: Wait for the verdict
        env:
          ROBORAMA_API_KEY: ${{ secrets.ROBORAMA_API_KEY }}
        run: |
          while true; do
            run=$(curl -sS "https://api.roborama.com/v1/runs?kind=eval&limit=1" \
              -H "Authorization: Bearer $ROBORAMA_API_KEY" | jq '.data[0]')
            status=$(echo "$run" | jq -r '.status')
            echo "suite run status: $status"
            case "$status" in
              completed)
                echo "$run" | jq '{result: .result, regressions: .regressions}'
                if [ "$(echo "$run" | jq '.regressions | length')" -eq 0 ]; then
                  exit 0
                fi
                exit 1
                ;;
              failed|stopped)
                exit 1
                ;;
              *)
                sleep 120
                ;;
            esac
          done
```

The polling step holds a runner while robots work through the suite. That is
fine for small gates; a full `rbr-manip-core@v3` pass takes hours of wall
clock and can outlast a hosted runner's six-hour cap. For those, point the
`run.completed` and `regression.detected` webhooks at a receiver that sets
the commit status instead — nothing waits, and the verdict still lands on
the pull request.

## The five webhook events

- `run.completed` — any run reached `completed`. The payload carries the full
  result: n, rate, CI, clusters, both meters, cost. The generic "verdict is
  in" signal.
- `episode.failed` — one failed episode, as it happens, with its `cluster`
  label and a video URL. Subscribe to triage while the suite is still
  running.
- `threshold.crossed` — a [`threshold()`](/docs/primitives/threshold/)
  contract met its target on the iterate tier; escalation to the verification
  tier is queued.
- `regression.detected` — a gate or matrix comparison found a cell below
  baseline. This is the one to wire to your release process.
- `estop.triggered` — an emergency stop fired in a cell running your policy.
  The run stops, the episode is marked `estop`, and the cell is inspected
  before resuming; partial results are kept.

A `regression.detected` delivery, verbatim:

```json
{
  "id": "evt_5150",
  "type": "regression.detected",
  "created": "2026-09-14T08:03:55Z",
  "data": {
    "run_id": "run_8901",
    "vs": "run_8821",
    "robot": "g1-edu-pro@fw2.4",
    "environment": "kitchen-std@v1.2",
    "delta_pts": -10.5,
    "cluster": "grasp_slip"
  }
}
```

Behind that delta: `kitchen-std@v1.2` on fw2.4 came in at 0.787 (n=240, ci95
0.731–0.835) against 0.892 (n=240, ci95 0.846–0.925) on fw2.3 in `run_8821`.
A wrist controller change opened a `grasp_slip` cluster — the webhook names
the cluster so triage starts at the right episodes, not at a diff of the
firmware release notes.

## Watch it live

Gate runs stream like any other run: an episode ticker plus WebRTC streams
for the cell doing the work.

*Example: watch it live*

**Python**

```python
import roborama  # reads ROBORAMA_API_KEY from the environment

run = roborama.runs.get("run_8842")

for event in run.watch():   # live: episode ticker + WebRTC stream URLs
    print(event.episode, event.status)

stream = roborama.streams.get("cell-g1-04")
print(stream)
# {webrtc: "wss://streams.roborama.com/cells/cell-g1-04/webrtc",
#  mjpeg: "https://streams.roborama.com/cells/cell-g1-04/mjpeg",
#  viewer_token: "vt_7Kq2mHentXw4"}
```

**TypeScript**

```typescript
import Roborama from "@roborama/sdk"; // reads ROBORAMA_API_KEY

const roborama = new Roborama();

const run = await roborama.runs.get("run_8842");

for await (const event of run.watch()) {
  // live: episode ticker + WebRTC stream URLs
  console.log(event.episode, event.status);
}

const stream = await roborama.streams.get("cell-g1-04");
console.log(stream);
// { webrtc: "wss://streams.roborama.com/cells/cell-g1-04/webrtc",
//   mjpeg: "https://streams.roborama.com/cells/cell-g1-04/mjpeg",
//   viewer_token: "vt_7Kq2mHentXw4" }
```

**cURL**

```bash
# live event stream (server-sent events): episode ticker + status
curl -N https://api.roborama.com/v1/runs/run_8842/events \
  -H "Authorization: Bearer $ROBORAMA_API_KEY"

# cell video stream descriptor
curl https://api.roborama.com/v1/streams/cell-g1-04 \
  -H "Authorization: Bearer $ROBORAMA_API_KEY"
```

**Agent (tool-use payload)**

```json
[
  {
    "type": "tool_use",
    "name": "roborama_watch_run",
    "input": { "run_id": "run_8842" }
  },
  {
    "type": "tool_use",
    "name": "roborama_get_stream",
    "input": { "cell": "cell-g1-04" }
  }
]
```