# compare()

Paired A/B on physical hardware — same initial conditions per episode pair, half the episodes to separate two policies.

"Is v4 better than v3" is a different question from "what is v4's rate" — and
it is cheaper to answer, if you run it right. `roborama.compare()` runs two
policies in paired episodes and returns a verdict with a p-value and an
interval on the effect, not just two rates to squint at.

## The call

*Example: roborama.compare()*

**Python**

```python
import roborama  # reads ROBORAMA_API_KEY from the environment

pol_a = roborama.Policy.checkpoint(hf="acme/skill-v3", runtime="openpi")
pol_b = roborama.Policy.checkpoint(hf="acme/skill-v4", runtime="openpi")

ab = roborama.compare(
    policies={"v3": pol_a, "v4": pol_b},
    paired=True,                # same initial conditions per episode pair —
                                # halves the n needed to separate them
    robot="g1-edu-pro@fw2.3",
    task="shelf_restock@v1",
)

print(ab.winner)   # "v4", p=0.008, effect=+4.1pts
```

**TypeScript**

```typescript
import Roborama from "@roborama/sdk"; // reads ROBORAMA_API_KEY

const roborama = new Roborama();

const ab = await roborama.runs.create({
  kind: "compare",
  policies: {
    v3: { type: "checkpoint", hf: "acme/skill-v3", runtime: "openpi" },
    v4: { type: "checkpoint", hf: "acme/skill-v4", runtime: "openpi" },
  },
  paired: true, // same initial conditions per episode pair —
  //              halves the n needed to separate them
  robot: "g1-edu-pro@fw2.3",
  task: "shelf_restock@v1",
});

console.log(ab.winner); // "v4", p=0.008, effect=+4.1pts
```

**cURL**

```bash
curl https://api.roborama.com/v1/runs \
  -H "Authorization: Bearer $ROBORAMA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "kind": "compare",
    "policies": {
      "v3": { "type": "checkpoint", "hf": "acme/skill-v3",
              "runtime": "openpi" },
      "v4": { "type": "checkpoint", "hf": "acme/skill-v4",
              "runtime": "openpi" }
    },
    "paired": true,
    "robot": "g1-edu-pro@fw2.3",
    "task": "shelf_restock@v1"
  }'
```

**Agent (tool-use payload)**

```json
{
  "type": "tool_use",
  "name": "roborama_compare",
  "input": {
    "policies": {
      "v3": { "type": "checkpoint", "hf": "acme/skill-v3",
              "runtime": "openpi" },
      "v4": { "type": "checkpoint", "hf": "acme/skill-v4",
              "runtime": "openpi" }
    },
    "paired": true,
    "robot": "g1-edu-pro@fw2.3",
    "task": "shelf_restock@v1"
  }
}
```

## Why pairing halves the n

In an unpaired A/B, each policy sees its own random draw of scenes. An
unlucky layout or a harsh lighting draw lands on one arm and not the other,
and that between-scene variance sits in your noise term — you buy extra
episodes just to average it out.

With `paired=True`, each episode pair replays the same initial conditions for
both policies: the same layout draw, the same lighting, the same distractor
placement, generated from the same seeded perturbation schedule that makes
[run()](/docs/primitives/run/) replayable. Whatever the scene contributes to
difficulty, it contributes to both arms of the pair equally — so when the
test looks at within-pair differences, the shared scene variance cancels.
What remains is the policy difference plus a much smaller residual, which is
why pairing needs roughly half the episodes to separate two policies at the
same power.

The pairs are also scheduled back-to-back on the same cell, so calibration
age and thermal state stay close within a pair; both are recorded per episode
in `/meta` if you want to check.

## Reading the verdict

The canonical result: winner `"v4"`, p=0.008, effect +4.1 pts (95% CI +1.1 to
+7.1 pts). Read the interval before the p-value. The entire interval is
positive, so v4 really is better; the lower end says the improvement is at
least about +1.1 pts, the pessimistic case you should plan around. When the
interval straddles zero, there is no winner to ship — `ab.winner` will tell
you so rather than flattering the point estimate.

## What compare() is not

Pairing sharpens the *difference*; it does not buy you a tighter absolute
rate for either policy. If the release question is "what is v4's rate in this
scene, with what precision," that is a [run()](/docs/primitives/run/) with
`episodes="auto(ci=0.95, moe=0.03)"`. Use `compare()` to pick the winner,
then `run()` or [matrix()](/docs/primitives/matrix/) to characterize it.

## Where next

- [Runs & statistics](/docs/concepts/runs-and-statistics/) — intervals, sizing, and why paired tests work.
- [transfer()](/docs/primitives/transfer/) — when the comparison is between embodiments, not policies.