The transfer gap

Measure what an embodiment change costs before your customer does, then close the gap cluster by cluster.

A policy that scores well on the embodiment it was trained on has told you nothing about the one your customer runs. transfer() measures the drop directly — same task, same perturbation schedule, source embodiment against target — so the gap is a number with an interval, not a hunch.

import roborama  # reads ROBORAMA_API_KEY from the environment

policy = roborama.Policy.checkpoint(hf="acme/skill-v4", runtime="openpi")

gap = roborama.transfer(
    policy=policy,
    source="aloha-bimanual@fw3.1",   # the embodiment it was trained on
    target="g1-edu-pro@fw2.3",
    task="fold_towel@v2",
)

print(gap.report())
# source: n=380  rate=0.942  ci95=(0.914, 0.961)
# target: n=340  rate=0.715  ci95=(0.665, 0.760)
# failure clusters opened in transfer: [grasp_slip, wrist_singularity]

Reading the report

On fold_towel@v2, the canonical policy scores 0.942 (n=380, ci95 0.914–0.961) on aloha-bimanual@fw3.1, the embodiment it was trained on. On g1-edu-pro@fw2.3 it scores 0.715 (n=340, ci95 0.665–0.760) — a 22.7-point gap, and the intervals are nowhere near overlapping. This is not noise.

The number says how much. The clusters say what: grasp_slip and wrist_singularity opened in transfer — failure modes the source embodiment never exhibited. A different gripper geometry slips where the source's didn't; the target's wrist reaches joint configurations the policy never had to steer around. That is a diagnosis, not just a grade, and it is where the playbook starts.

The playbook

1. Read the clusters first

   They partition the gap into causes you can act on. A gap that is mostly grasp_slip is usually a data problem; wrist_singularity points at a contract or retargeting problem.

2. Pull the failing episodes

   Filter episode artifacts by cluster and read the evidence: video for the what, MCAP for the why — /joint_states_measured against /joint_targets_commanded shows exactly where execution diverged from intent.

3. Fix what broke

   Fine-tune on target-embodiment episodes (run.export(format="lerobot") feeds most training stacks), or fix the declared contract — an action_space or observation_contract mismatch produces exactly these clusters. See policies.

4. Re-measure

   Re-run transfer() with the same pins. For before/after fine-tune deltas, compare() with paired=True runs both checkpoints against the same initial conditions per episode pair — pairing halves the n you need to separate them.

5. Pin it so it can't reopen

   Once the gap closes, add the target embodiment as a row in your matrix() and to a CI gate. A transfer gap you closed once is a regression waiting for a firmware bump.

Measure before your customer does. The gap does not care which of you finds it first, but your release process should.