# Evaluate a custom policy: code on GitHub, weights in private storage

CI assembles a container from a GitHub repo and private S3 weights, pins its digest, and submits 600 humanoid episodes under a $4,000 cap.

*Provider workflow: GitHub, Amazon S3, Docker.*

Not every policy is a checkpoint a managed runtime can load. This one is
custom: the inference server lives in a GitHub repo, the weights sit in
a private S3 bucket, and there is no `s3://` checkpoint source or
storage-credential field to point at them. Neither needs to be
importable. A GitHub Actions job assembles a container, pushes it to
`ghcr.io`, and Roborama runs the container — the credential that can
read your bucket exists for the duration of one CI job and never
reaches us.

### Lay out the pieces

```text
github.com/acme/skill
├── server/serve.py                # the inference server
├── server/policy.py
├── requirements.txt               # pinned dependencies
├── Dockerfile
└── .github/workflows/evaluate.yml

s3://acme-checkpoints/skill/v4/
├── model.safetensors
├── vision_encoder.safetensors
└── manifest.txt                   # every file + its sha256
```

The server implements the cell interface: it consumes the observations
named by `droid-3cam`, emits `joint_delta_50hz` actions at the declared
rate, and clears its per-episode state at each episode boundary — scene
reset is the cell's job, never the policy's.
[Run your container](/docs/guides/run-your-container/) covers the
interface;
[Import from private storage](/docs/guides/import-from-private-storage/)
covers why the bucket stays private.

The manifest is what makes the weights auditable: CI verifies every
file against its sha256 before anything goes into an image, so "which
weights are in production" always has a checkable answer.

### Write the Dockerfile

CI downloads the verified weights into `weights/` before the build, so
the image is self-contained — weights baked in, credentials not:

```dockerfile
FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY server/ server/

# weights/ is populated by CI before `docker build` runs;
# nothing in the image can read the bucket
COPY weights/ /app/weights/
ENV SKILL_WEIGHTS_DIR=/app/weights

ENTRYPOINT ["python", "-m", "server.serve"]
```

### Let CI assemble, verify, and submit

One `workflow_dispatch` workflow does the whole assembly: assume a
scoped AWS role via OIDC (no long-lived key in secrets), fetch and
checksum the weights, build and push the image, capture its digest,
then quote and submit. The conventions from
[CI gates](/docs/guides/ci-gates/) carry over unchanged: the API key
arrives as `secrets.ROBORAMA_API_KEY`, and the submission carries a
hard `max_budget_usd`.

```yaml
name: evaluate-on-roborama

on:
  workflow_dispatch:
    inputs:
      weights_version:
        description: "Prefix under s3://acme-checkpoints"
        default: skill/v4

permissions:
  contents: read
  packages: write     # push to ghcr.io
  id-token: write     # OIDC — no long-lived AWS key in secrets

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Assume the scoped S3 read role (OIDC)
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: >-
            arn:aws:iam::123456789012:role/rbr-weights-read
          aws-region: us-east-1

      - name: Fetch and verify the weights
        run: |
          aws s3 cp --recursive \
            "s3://acme-checkpoints/${{ inputs.weights_version }}/" \
            weights/
          cd weights && sha256sum -c manifest.txt

      - name: Log in to ghcr.io
        run: |
          echo "${{ secrets.GITHUB_TOKEN }}" \
            | docker login ghcr.io -u "${{ github.actor }}" \
                --password-stdin

      - name: Build, push, capture the digest
        id: build
        run: |
          docker build -t ghcr.io/acme/skill:v4 .
          docker push ghcr.io/acme/skill:v4
          digest=$(docker inspect ghcr.io/acme/skill:v4 \
            --format '{{ index .RepoDigests 0 }}')
          echo "digest=${digest#*@}" >> "$GITHUB_OUTPUT"

      - name: Quote (free — both meters, before anything books)
        env:
          ROBORAMA_API_KEY: ${{ secrets.ROBORAMA_API_KEY }}
        run: |
          curl -sS --fail https://api.roborama.com/v1/quote \
            -H "Authorization: Bearer $ROBORAMA_API_KEY" \
            -H "Content-Type: application/json" \
            -d '{
              "robot": "g1-edu-pro",
              "environment": "kitchen-std",
              "episodes": 600
            }'

      - name: Submit the run
        env:
          ROBORAMA_API_KEY: ${{ secrets.ROBORAMA_API_KEY }}
        run: |
          # pin the exact image this job built — commit
          # ${{ github.sha }} is what gets evaluated
          img="ghcr.io/acme/skill@${{ steps.build.outputs.digest }}"
          run_id=$(curl -sS --fail \
            https://api.roborama.com/v1/runs \
            -H "Authorization: Bearer $ROBORAMA_API_KEY" \
            -H "Content-Type: application/json" \
            -d '{
              "kind": "run",
              "robot": "g1-edu-pro@fw2.3",
              "environment": "kitchen-std@v1.2",
              "policy": {
                "type": "container",
                "image": "'"$img"'",
                "action_space": "joint_delta_50hz",
                "observation_contract": "droid-3cam"
              },
              "task": "load_dishwasher@v2",
              "episodes": "auto(ci=0.95, moe=0.03)",
              "max_budget_usd": 4000
            }' | jq -r '.id')
          echo "run=$run_id commit=${{ github.sha }}" \
            >> "$GITHUB_STEP_SUMMARY"
```

The role behind `rbr-weights-read` grants `s3:GetObject` on
`acme-checkpoints/skill/*` and nothing else, and the assumed credential
lives and dies inside the job. One visibility check before the first
dispatch: the facility pulls `ghcr.io/acme/skill` anonymously — ghcr
packages default to private, so flip the package to public (registry
pull credentials are not supported today).

### Pin the digest, buy the episodes

`ghcr.io/acme/skill:v4` is a tag, and tags move. The digest names the
exact bytes CI just built — the same reflex as pinning
`g1-edu-pro@fw2.3` instead of "whatever firmware was around", and it is
what the verification report should cite. Outside CI — from a
workstation, or an agent — the same submission looks like this:

*Example: pin the digest, buy the episodes*

**Python**

```python
import roborama  # reads ROBORAMA_API_KEY from the environment

# The digest CI captured at push — the exact image, not a tag
image = (
    "ghcr.io/acme/skill@sha256:"
    "4e5b1a7c9d0f2a3b5c6d8e9f0a1b2c3d"
    "4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a90"
)

policy = roborama.Policy.container(
    image=image,
    action_space="joint_delta_50hz",
    observation_contract="droid-3cam",
)

run = roborama.run(
    robot="g1-edu-pro@fw2.3",
    environment="kitchen-std@v1.2",
    policy=policy,
    task="load_dishwasher@v2",
    episodes="auto(ci=0.95, moe=0.03)",
    max_budget_usd=4000,
)

# Later — when the run completes
print(run.result())
# n=612  rate=0.874  ci95=(0.846, 0.898)
# failure_clusters: {grasp_slip: 41, perception_miss: 22,
#                    collision: 9}
# robot_hours=20.4  environment_hours=20.4  cost_usd=3812

run.report(format="pdf")  # the citable verification report
```

**TypeScript**

```typescript
import Roborama from "@roborama/sdk"; // reads ROBORAMA_API_KEY

const roborama = new Roborama();

// The digest CI captured at push — the exact image, not a tag
const image =
  "ghcr.io/acme/skill@sha256:" +
  "4e5b1a7c9d0f2a3b5c6d8e9f0a1b2c3d" +
  "4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a90";

const run = await roborama.runs.create({
  robot: "g1-edu-pro@fw2.3",
  environment: "kitchen-std@v1.2",
  policy: {
    type: "container",
    image,
    action_space: "joint_delta_50hz",
    observation_contract: "droid-3cam",
  },
  task: "load_dishwasher@v2",
  episodes: "auto(ci=0.95, moe=0.03)",
  max_budget_usd: 4000,
});

// Later — when the run completes
console.log(await run.result());
// n=612  rate=0.874  ci95=(0.846, 0.898)
// failure_clusters: { grasp_slip: 41, perception_miss: 22,
//                     collision: 9 }
// robot_hours=20.4  environment_hours=20.4  cost_usd=3812

await run.report({ format: "pdf" }); // verification report
```

**cURL**

```bash
# The digest CI captured at push — the exact image, not a tag
IMG="ghcr.io/acme/skill@sha256:"
IMG="${IMG}4e5b1a7c9d0f2a3b5c6d8e9f0a1b2c3d"
IMG="${IMG}4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a90"

curl -sS https://api.roborama.com/v1/runs \
  -H "Authorization: Bearer $ROBORAMA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "kind": "run",
    "robot": "g1-edu-pro@fw2.3",
    "environment": "kitchen-std@v1.2",
    "policy": {
      "type": "container",
      "image": "'"$IMG"'",
      "action_space": "joint_delta_50hz",
      "observation_contract": "droid-3cam"
    },
    "task": "load_dishwasher@v2",
    "episodes": "auto(ci=0.95, moe=0.03)",
    "max_budget_usd": 4000
  }'

# Later — the result and the verification report
curl -sS https://api.roborama.com/v1/runs/run_8842 \
  -H "Authorization: Bearer $ROBORAMA_API_KEY"
# n=612 rate=0.874 ci95=(0.846, 0.898) cost_usd=3812

curl -sS -o report.pdf \
  "https://api.roborama.com/v1/runs/run_8842/report?format=pdf" \
  -H "Authorization: Bearer $ROBORAMA_API_KEY"
```

**Agent (tool-use payload)**

```json
[
  {
    "type": "tool_use",
    "name": "roborama_run",
    "input": {
      "robot": "g1-edu-pro@fw2.3",
      "environment": "kitchen-std@v1.2",
      "policy": {
        "type": "container",
        "image": "ghcr.io/acme/skill@sha256:4e5b1a7c9d0f2a3b5c6d8e9f0a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a90",
        "action_space": "joint_delta_50hz",
        "observation_contract": "droid-3cam"
      },
      "task": "load_dishwasher@v2",
      "episodes": "auto(ci=0.95, moe=0.03)",
      "max_budget_usd": 4000
    }
  },
  {
    "type": "tool_use",
    "name": "roborama_get_report",
    "input": { "run_id": "run_8842", "format": "pdf" }
  }
]
```

`episodes="auto(ci=0.95, moe=0.03)"` sizes the run to a ±3-point margin
of error instead of a fixed n. Admission validates the declared
contracts against the pinned robot and scene before any motor moves —
a rejected submission returns `contract_validation_failed` and costs
nothing.

### What the quote comes back as

Back in the Actions log, the quote step printed the price of asking:
600 humanoid episodes in a replica kitchen, both meters visible:

```json
{
  "object": "quote",
  "robot": "g1-edu-pro",
  "environment": "kitchen-std",
  "episodes": 600,
  "priority": "standard",
  "robot_hours": 20.0,
  "env_hours": 20.0,
  "usd": 3740,
  "queue_eta": "6h",
  "breakdown": {
    "minutes_per_episode_est": 2.0,
    "robot_usd_per_hour": 148,
    "env_usd_per_hour": 39,
    "robot_usd": 2960,
    "env_usd": 780
  },
  "expires": "2026-09-08T00:00:00Z"
}
```

> **Fixture output:** The quote and run on this page — $3,740 for 600 episodes, `run_8842`
> and its statistics — are the canonical documentation fixtures from
> [/fixtures/runs.json](/fixtures/runs.json): internally consistent
> numbers, not live API responses.

Nothing has billed yet — quotes are free, rejected submissions are
free. The meter starts when a motor moves.

### Read the result, keep the evidence

`run_8842`, complete:

```json
{
  "n": 612,
  "success_rate": 0.874,
  "ci95": [0.846, 0.898],
  "interval_method": "wilson",
  "failure_clusters": {
    "grasp_slip": 41,
    "perception_miss": 22,
    "collision": 9
  },
  "robot_hours": 20.4,
  "environment_hours": 20.4,
  "cost_usd": 3812
}
```

The auto-sized run settled at n=612 — the margin-of-error target, not
the quote's 600, decided when to stop — and the meters billed what ran:
20.4 hours each, $3,812 against the $3,740 estimate, well inside the
$4,000 cap. 0.874 with a Wilson interval of (0.846, 0.898) is a number
a release review can lean on, and the clusters say where the other
episodes went.

`run.report(format="pdf")` renders the verification report — rate,
interval, resolved pins, the image digest among them. `run.episodes`
holds the per-episode evidence: video, MCAP telemetry, and a replay
spec for each episode, so every `collision` is a click away. And the
workflow's last line wrote the run id and `github.sha` to the job
summary — the result is traceable to the commit that produced it.

### When v5 ships

Same `workflow_dispatch`, `weights_version: skill/v5`, a new digest —
nothing else changes. To separate v5 from v4 rather than re-measure it,
hand both images to
[`roborama.compare()`](/docs/primitives/compare/): paired episodes
replay the same initial conditions for both, so the verdict arrives in
about half the episodes of two independent runs.

## Where next

- [Run your container](/docs/guides/run-your-container/) — the cell interface your server implements, in detail.
- [Package code as a container](/docs/guides/package-your-policy/) — the container path without the CI framing.
- [CI gates](/docs/guides/ci-gates/) — turn this workflow into a release check that fails on regression.
- [compare()](/docs/primitives/compare/) — v5 against v4, paired.