Get a demoSign up
Documentation menu

Getting started

Concepts

Bring your policy

Primitives

API reference

SDKs

Guides

Reference

Evaluate a custom policy: code on GitHub, weights in private storage

CI assembles a container from a GitHub repo and private S3 weights, pins its digest, and submits 600 humanoid episodes under a $4,000 cap.

GitHubAmazon S3Docker

Not every policy is a checkpoint a managed runtime can load. This one is custom: the inference server lives in a GitHub repo, the weights sit in a private S3 bucket, and there is no s3:// checkpoint source or storage-credential field to point at them. Neither needs to be importable. A GitHub Actions job assembles a container, pushes it to ghcr.io, and Roborama runs the container — the credential that can read your bucket exists for the duration of one CI job and never reaches us.

  1. Lay out the pieces
    github.com/acme/skill
├── server/serve.py                # the inference server
├── server/policy.py
├── requirements.txt               # pinned dependencies
├── Dockerfile
└── .github/workflows/evaluate.yml

s3://acme-checkpoints/skill/v4/
├── model.safetensors
├── vision_encoder.safetensors
└── manifest.txt                   # every file + its sha256

    The server implements the cell interface: it consumes the observations named by droid-3cam, emits joint_delta_50hz actions at the declared rate, and clears its per-episode state at each episode boundary — scene reset is the cell's job, never the policy's. Run your container covers the interface; Import from private storage covers why the bucket stays private.

    The manifest is what makes the weights auditable: CI verifies every file against its sha256 before anything goes into an image, so "which weights are in production" always has a checkable answer.

  2. Write the Dockerfile

    CI downloads the verified weights into weights/ before the build, so the image is self-contained — weights baked in, credentials not:

    FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY server/ server/

# weights/ is populated by CI before `docker build` runs;
# nothing in the image can read the bucket
COPY weights/ /app/weights/
ENV SKILL_WEIGHTS_DIR=/app/weights

ENTRYPOINT ["python", "-m", "server.serve"]
  3. Let CI assemble, verify, and submit

    One workflow_dispatch workflow does the whole assembly: assume a scoped AWS role via OIDC (no long-lived key in secrets), fetch and checksum the weights, build and push the image, capture its digest, then quote and submit. The conventions from CI gates carry over unchanged: the API key arrives as secrets.ROBORAMA_API_KEY, and the submission carries a hard max_budget_usd.

    name: evaluate-on-roborama

on:
  workflow_dispatch:
    inputs:
      weights_version:
        description: "Prefix under s3://acme-checkpoints"
        default: skill/v4

permissions:
  contents: read
  packages: write     # push to ghcr.io
  id-token: write     # OIDC — no long-lived AWS key in secrets

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Assume the scoped S3 read role (OIDC)
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: >-
            arn:aws:iam::123456789012:role/rbr-weights-read
          aws-region: us-east-1

      - name: Fetch and verify the weights
        run: |
          aws s3 cp --recursive \
            "s3://acme-checkpoints/${{ inputs.weights_version }}/" \
            weights/
          cd weights && sha256sum -c manifest.txt

      - name: Log in to ghcr.io
        run: |
          echo "${{ secrets.GITHUB_TOKEN }}" \
            | docker login ghcr.io -u "${{ github.actor }}" \
                --password-stdin

      - name: Build, push, capture the digest
        id: build
        run: |
          docker build -t ghcr.io/acme/skill:v4 .
          docker push ghcr.io/acme/skill:v4
          digest=$(docker inspect ghcr.io/acme/skill:v4 \
            --format '{{ index .RepoDigests 0 }}')
          echo "digest=${digest#*@}" >> "$GITHUB_OUTPUT"

      - name: Quote (free — both meters, before anything books)
        env:
          ROBORAMA_API_KEY: ${{ secrets.ROBORAMA_API_KEY }}
        run: |
          curl -sS --fail https://api.roborama.com/v1/quote \
            -H "Authorization: Bearer $ROBORAMA_API_KEY" \
            -H "Content-Type: application/json" \
            -d '{
              "robot": "g1-edu-pro",
              "environment": "kitchen-std",
              "episodes": 600
            }'

      - name: Submit the run
        env:
          ROBORAMA_API_KEY: ${{ secrets.ROBORAMA_API_KEY }}
        run: |
          # pin the exact image this job built — commit
          # ${{ github.sha }} is what gets evaluated
          img="ghcr.io/acme/skill@${{ steps.build.outputs.digest }}"
          run_id=$(curl -sS --fail \
            https://api.roborama.com/v1/runs \
            -H "Authorization: Bearer $ROBORAMA_API_KEY" \
            -H "Content-Type: application/json" \
            -d '{
              "kind": "run",
              "robot": "g1-edu-pro@fw2.3",
              "environment": "kitchen-std@v1.2",
              "policy": {
                "type": "container",
                "image": "'"$img"'",
                "action_space": "joint_delta_50hz",
                "observation_contract": "droid-3cam"
              },
              "task": "load_dishwasher@v2",
              "episodes": "auto(ci=0.95, moe=0.03)",
              "max_budget_usd": 4000
            }' | jq -r '.id')
          echo "run=$run_id commit=${{ github.sha }}" \
            >> "$GITHUB_STEP_SUMMARY"

    The role behind rbr-weights-read grants s3:GetObject on acme-checkpoints/skill/* and nothing else, and the assumed credential lives and dies inside the job. One visibility check before the first dispatch: the facility pulls ghcr.io/acme/skill anonymously — ghcr packages default to private, so flip the package to public (registry pull credentials are not supported today).

  4. Pin the digest, buy the episodes

    ghcr.io/acme/skill:v4 is a tag, and tags move. The digest names the exact bytes CI just built — the same reflex as pinning g1-edu-pro@fw2.3 instead of "whatever firmware was around", and it is what the verification report should cite. Outside CI — from a workstation, or an agent — the same submission looks like this:

    pin the digest, buy the episodes
    import roborama  # reads ROBORAMA_API_KEY from the environment

# The digest CI captured at push — the exact image, not a tag
image = (
    "ghcr.io/acme/skill@sha256:"
    "4e5b1a7c9d0f2a3b5c6d8e9f0a1b2c3d"
    "4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a90"
)

policy = roborama.Policy.container(
    image=image,
    action_space="joint_delta_50hz",
    observation_contract="droid-3cam",
)

run = roborama.run(
    robot="g1-edu-pro@fw2.3",
    environment="kitchen-std@v1.2",
    policy=policy,
    task="load_dishwasher@v2",
    episodes="auto(ci=0.95, moe=0.03)",
    max_budget_usd=4000,
)

# Later — when the run completes
print(run.result())
# n=612  rate=0.874  ci95=(0.846, 0.898)
# failure_clusters: {grasp_slip: 41, perception_miss: 22,
#                    collision: 9}
# robot_hours=20.4  environment_hours=20.4  cost_usd=3812

run.report(format="pdf")  # the citable verification report

    episodes="auto(ci=0.95, moe=0.03)" sizes the run to a ±3-point margin of error instead of a fixed n. Admission validates the declared contracts against the pinned robot and scene before any motor moves — a rejected submission returns contract_validation_failed and costs nothing.

  5. What the quote comes back as

    Back in the Actions log, the quote step printed the price of asking: 600 humanoid episodes in a replica kitchen, both meters visible:

    {
  "object": "quote",
  "robot": "g1-edu-pro",
  "environment": "kitchen-std",
  "episodes": 600,
  "priority": "standard",
  "robot_hours": 20.0,
  "env_hours": 20.0,
  "usd": 3740,
  "queue_eta": "6h",
  "breakdown": {
    "minutes_per_episode_est": 2.0,
    "robot_usd_per_hour": 148,
    "env_usd_per_hour": 39,
    "robot_usd": 2960,
    "env_usd": 780
  },
  "expires": "2026-09-08T00:00:00Z"
}

    Nothing has billed yet — quotes are free, rejected submissions are free. The meter starts when a motor moves.

  6. Read the result, keep the evidence

    run_8842, complete:

    {
  "n": 612,
  "success_rate": 0.874,
  "ci95": [0.846, 0.898],
  "interval_method": "wilson",
  "failure_clusters": {
    "grasp_slip": 41,
    "perception_miss": 22,
    "collision": 9
  },
  "robot_hours": 20.4,
  "environment_hours": 20.4,
  "cost_usd": 3812
}

    The auto-sized run settled at n=612 — the margin-of-error target, not the quote's 600, decided when to stop — and the meters billed what ran: 20.4 hours each, $3,812 against the $3,740 estimate, well inside the $4,000 cap. 0.874 with a Wilson interval of (0.846, 0.898) is a number a release review can lean on, and the clusters say where the other episodes went.

    run.report(format="pdf") renders the verification report — rate, interval, resolved pins, the image digest among them. run.episodes holds the per-episode evidence: video, MCAP telemetry, and a replay spec for each episode, so every collision is a click away. And the workflow's last line wrote the run id and github.sha to the job summary — the result is traceable to the commit that produced it.

  7. When v5 ships

    Same workflow_dispatch, weights_version: skill/v5, a new digest — nothing else changes. To separate v5 from v4 rather than re-measure it, hand both images to roborama.compare(): paired episodes replay the same initial conditions for both, so the verdict arrives in about half the episodes of two independent runs.

Where next