compare()

Paired A/B on physical hardware — same initial conditions per episode pair, half the episodes to separate two policies.

"Is v4 better than v3" is a different question from "what is v4's rate" — and it is cheaper to answer, if you run it right. roborama.compare() runs two policies in paired episodes and returns a verdict with a p-value and an interval on the effect, not just two rates to squint at.

The call

import roborama  # reads ROBORAMA_API_KEY from the environment

pol_a = roborama.Policy.checkpoint(hf="acme/skill-v3", runtime="openpi")
pol_b = roborama.Policy.checkpoint(hf="acme/skill-v4", runtime="openpi")

ab = roborama.compare(
    policies={"v3": pol_a, "v4": pol_b},
    paired=True,                # same initial conditions per episode pair —
                                # halves the n needed to separate them
    robot="g1-edu-pro@fw2.3",
    task="shelf_restock@v1",
)

print(ab.winner)   # "v4", p=0.008, effect=+4.1pts

Why pairing halves the n

In an unpaired A/B, each policy sees its own random draw of scenes. An unlucky layout or a harsh lighting draw lands on one arm and not the other, and that between-scene variance sits in your noise term — you buy extra episodes just to average it out.

With paired=True, each episode pair replays the same initial conditions for both policies: the same layout draw, the same lighting, the same distractor placement, generated from the same seeded perturbation schedule that makes run() replayable. Whatever the scene contributes to difficulty, it contributes to both arms of the pair equally — so when the test looks at within-pair differences, the shared scene variance cancels. What remains is the policy difference plus a much smaller residual, which is why pairing needs roughly half the episodes to separate two policies at the same power.

The pairs are also scheduled back-to-back on the same cell, so calibration age and thermal state stay close within a pair; both are recorded per episode in /meta if you want to check.

Reading the verdict

The canonical result: winner "v4", p=0.008, effect +4.1 pts (95% CI +1.1 to +7.1 pts). Read the interval before the p-value. The entire interval is positive, so v4 really is better; the lower end says the improvement is at least about +1.1 pts, the pessimistic case you should plan around. When the interval straddles zero, there is no winner to ship — ab.winner will tell you so rather than flattering the point estimate.

What compare() is not

Pairing sharpens the difference; it does not buy you a tighter absolute rate for either policy. If the release question is "what is v4's rate in this scene, with what precision," that is a run() with episodes="auto(ci=0.95, moe=0.03)". Use compare() to pick the winner, then run() or matrix() to characterize it.

Where next

• Runs & statistics — intervals, sizing, and why paired tests work.
• transfer() — when the comparison is between embodiments, not policies.