# Research The evidence behind the product: why learned policies can only be verified empirically, what simulation can and cannot carry, and the arithmetic that sets the sample sizes. Sources at the bottom; the full treatment is in the white paper. ## 1. Why robot policies cannot be reviewed — only measured Classical software earns trust through inspection: code review, unit tests, formal verification. A modern robot policy forecloses that path by construction. It is not a program but a set of learned weights — billions of parameters fine-tuned from demonstrations — and weights cannot be read. There is no line of code that says what the policy does when the cup is 40 mm left of its training distribution; there is only what it *does*. The industry has replaced legible programs with behavioral black boxes, and a black box admits exactly one verification method: run it, many times, under controlled conditions, and count. This is the paradigm shift the tooling has not caught up with. The software world built a $250B+ testing and assurance industry around *inspectable* artifacts. Robotics is shipping *uninspectable* ones into physical space — around people, at reach, with force — with an evidentiary standard of 20 to 30 hand-reset trials and a demo reel. Every section below follows from this one fact: **empiricism is not one option for verifying learned policies; it is the only one.** > Artifact: a policy checkpoint is 2–8 GB of `model.safetensors`. `grep` it > for the failure mode. (You can't. That's the point.) ## 2. The verification arithmetic Three pieces of boring, load-bearing statistics. **Reliability compounds — and it compounds against you.** Useful physical work is a chain of steps. A policy that succeeds 99% per step completes a 30-step task 74% of the time; at 95% per step, 21%. The entire economic value of deployment lives in the last decimals of per-step reliability — which is why demo reels (one take, survivor-selected) and small-n evals (20 trials cannot distinguish 85% from 99%) carry no information about deployment. **Demonstrating those decimals has a price set by mathematics, not by us.** By the rule of three, showing a failure rate below 1-in-1,000 at 95% confidence requires roughly 3,000 clean episodes; 1-in-10,000 requires 30,000. At 2–4 minutes per episode, one cell of the compatibility matrix — one task, one embodiment, one environment, one firmware — costs 100 to 200 robot-hours to verify at deployment grade. This is the floor under physical evaluation that no simulator can erode, because it is a property of binomial evidence, not of tooling. **The intervals are chosen for where policies actually live.** Success rates are binomial; we report Wilson score intervals because they behave correctly near 0 and 1 — the regions that matter — where the naive normal approximation fails. Sample size is an input: `episodes="auto(ci=0.95, moe=0.03)"` solves for n before the run starts. Paired designs (`compare(paired=True)`) reuse identical initial conditions across two policies and roughly halve the n needed to separate them. Any success rate published without n and an interval is treated — on this site and in the API — as a bug. > Artifact: 0.99³⁰ = 0.74. 0.95³⁰ = 0.21. The gap between a demo and a > deployment is thirty exponents wide. ## 3. What simulation can and cannot carry We state the strongest case against our own business, because it is also our largest future customer segment. The simulation stack is three technologies with three different boundaries: **Physics engines** (Isaac, MuJoCo, Genesis) compute the world from hand-written equations. They are the reason humanoid locomotion is a solved transfer story — massively parallel RL with domain randomization trained the sprinters. Their boundary is what the equations don't capture: contact-rich manipulation, deformables, visual fidelity. **Learned world models** (the Veeda thesis; Runway's robotics work) learn dynamics from video and generalize where equations cannot. Runway reports policy-evaluation correlation of r = 0.95 across eight policies without scene-specific reconstruction [4]. **Real-to-sim evaluators** are the newest and most consequential class. RoboWorld replays an entire real-world benchmark inside a neural simulator seeded with real initial observations, replicating the RoboArena leaderboard across eight policies at Pearson r = 0.989 for roughly 100 H100 GPU-hours [3]. PolaRiS reconstructs interactive scenes from short video scans and validates against 600 real rollouts [2]. If the question is *which of two policies is better on average*, neural evaluation is close to answering it without a robot — and improving fast. The same literature states the boundary with equal clarity: - These are **ranking correlations across a handful of policies, not calibrated absolute failure rates** — and deployment decisions live in the absolute decimals a ranking cannot supply (§2). - The correlations degrade at the edges: PolaRiS's worst single-environment correlation falls to **0.81** [2]. - Classical sim benchmarks actively mislead: nearly all modern policies score 90–95% in Libero while spanning the full spectrum on real hardware [2]. - Open video models **hallucinate during object interaction**, producing outright policy mis-rankings [2]. - Tail failures are out-of-distribution for any learned simulator **by construction** — a model cannot flag what its training distribution never contained. And the deepest structural fact: **the best evaluators are built from real data.** PolaRiS needs real demonstrations to close its residual gap; RoboWorld is seeded with real episodes. The better neural evaluation gets, the more calibrated physical ground truth it consumes. Aviation already ran this experiment to completion: Level-D flight simulators absorbed most training hours decades ago, yet each simulator must itself be qualified against objective flight-test data under FAA Part 60 — simulation became physical testing's largest *customer*, not its replacement [5]. That is the relationship [`verify()`](/docs/primitives/verify/) is shaped around, and it is why simulator vendors can license our ground truth under `sim-calibration-v1`. > Artifact: r = 0.989 for ranking [3]; 0.81 in the worst environment [2]; > undefined in the tail. The division of labor writes itself. ## 4. Locomotion is solved in sim. Manipulation is not. The Games proved both. At the second World Humanoid Robot Games (Beijing, August 2026 — 2,000+ robots, 600+ teams), a humanoid ran 100 m in 9.39 s, beating the human world record and roughly doubling the prior year's pace. In the same week's manipulation and contact events, competitors tripped, collapsed, caught fire, and — canonically — sprinters could not stop after the finish line, hitting the barriers at full speed [6]. Read as a controlled experiment, the Games separate the two halves of the field precisely. Locomotion, the discipline simulation trains best (parallelizable, contact-simple, reward-dense), is improving on a steep curve. Manipulation, recovery, and knowing-when-to-stop — the disciplines that require real interaction data — remain visibly unreliable. And the failure that matters most is the sprinter's: a policy that performs its trained behavior perfectly and cannot handle the boundary of that behavior is exactly the tail-event class that no training distribution contains and no learned simulator can flag (§3). The finish line is out-of-distribution. > Artifact: 9.39 s over 100 m — then a wall. Both facts in one run [6]. ## 5. Reset economics: seven of eight minutes **(Attribution corrected — this is a published Physical Intelligence result, not our study.)** In PI's March 2026 online-RL work, training a precision manipulation skill took two hours of wall-clock time for **fifteen minutes of robot data — the balance consumed by resets and overhead** [7]. At the best-resourced laboratory in the field, roughly seven of every eight minutes of a real-world RL run are reset economics, not robot learning. This is the finding the facility is organized around. Georgia Tech's Robotarium reached the same conclusion from the operations side: at roughly 500 experiments a month, manual reset became untenable and the facility automated wake-up, execution, diagnostics and shutdown — and has since run 16,500+ remote experiments on that architecture [8]. Resets are not overhead to be minimized ad hoc; they are *the* production function. It is why environments ship with declared reset methods, why reset time is a tracked KPI on every episode, and why the second meter on the bill is the [environment-hour](/pricing/): the scene, its fixtures, and its reset engineering are where the cost — and the compounding operational advantage — actually live. > Artifact: 15 minutes of robot data in a 120-minute run [7]. The other 105 > minutes are the business. ## 6. The testing surface compounds — a projection you can check Demand for physical verification is the product of four factors, each independently observable today: | Driver | 2026 | 2030 | 2035 (proj.) | | --- | --- | --- | --- | | Embodiment models in commercial service | ~15 | ~40 | ~80 | | Hardware revisions per model in service | 2 | 3 | 3–4 | | Cross-embodiment model families | ~6 | ~12 | ~20 | | Major releases per family per year | 3–4 | 4–6 | 6+ | | **Implied matrix cells needing verification / yr** | **~1–2k** | **~15–30k** | **~100k+** | Anchors: 25+ humanoid platforms shipping or announced; Goldman Sachs projects 1.4M annual humanoid shipments by 2035 [9]; the model layer has consolidated on explicitly cross-embodiment claims — every one a promise that must be re-verified per body. A fourth axis is forming above the models: **applications**. If robot skills distribute through app-store-like channels (the first consumer skills marketplaces are already live), every skill must be verified on every device it targets — exactly as mobile software is tested across handset matrices today. Robotics companies will not own fifty robots each in order to do it. Two properties matter more than the size. The surface compounds *multiplicatively* while any team's owned hardware grows linearly — the gap between what must be tested and what can be owned widens mechanically with industry success. And it arrives in overlapping phases per capability and vertical (logistics deploys years before general manipulation), so verification demand stacks vertical by vertical rather than waiting for a distant general-purpose future [10]. > Artifact: the bottom row of the table is a multiplication you can redo > yourself. We publish the factors so you can disagree with them precisely. ## 7. The transfer gap is measurable — and structural The field's training economics guarantee the gap. The leading foundation models train overwhelmingly on arm and wheeled-bimanual platforms — π0's published platform list is eight variations of arms and mobile bases, zero legged humanoids [11] — because arms maximize episodes-per-dollar. Every "one brain, many bodies" claim therefore ships with an unverified transfer to the bodies that matter most: balance-coupled, whole-body, differently sensed humanoids. The gap is not an argument against transfer; it is an argument for measuring it before your customer does. A representative [`transfer()`](/docs/primitives/transfer/) result for a towel-fold policy trained on `aloha-bimanual@fw3.1`: **0.942** on the source embodiment (n=380, ci95 0.914–0.961) falling to **0.715** on `g1-edu-pro@fw2.3` (n=340, ci95 0.665–0.760) — with failure clusters (`grasp_slip`, `wrist_singularity`) that did not exist at the source. The number nobody in the market can currently produce is the one every deployment decision needs. > Artifact: −22.7 points, with intervals on both ends and named clusters in > between. ## 8. Where verification sits when the industry matures Economies pay a predictable fraction of their value to the institutions that verify them. The global testing, inspection and certification industry runs at roughly a quarter-trillion dollars a year — about 0.2% of world output — and its largest players estimate outsourced services are only ~40% of total assurance spending [12]. Where the object under test is novel and safety-critical, the ratio runs higher: pharmaceutical companies outsource on the order of $85B annually to contract research organizations, a quarter or more of the industry's R&D [13]. Aviation, again, shows the equilibrium with simulation included: sim absorbs the volume, physical testing anchors the evidence, and the simulator itself is a customer of the flight-test data [5]. Applied to published robot-economy forecasts ($1T+ by 2040; up to $5T by 2050), historical capture ratios of 0.2–1% imply a physical-AI assurance layer of roughly **$10–60B per year at maturity** — a layer that grows with the *installed base*, independent of R&D budgets, because every model update pushed to a deployed fleet reopens the matrix. Verification is the workload that expands, rather than shrinks, as the industry approaches deployment. > Artifact: TIC ≈ $250B/yr on the physical economy [12]; CROs ≈ $85B/yr on > pharma R&D [13]. The robot economy will not be the first to skip the toll. ## 9. Sources 1. ArmBench and perturbation literature on evaluation sample sizes and unreported experimental choices in published robot evaluations. 2. PolaRiS (RSS 2026): real-to-sim evaluation via Gaussian-splat reconstruction; 600 real / 93,000 simulated rollouts; worst single-environment r = 0.81; Libero saturation; video-model hallucination findings. 3. RoboWorld (2026): neural replication of the RoboArena leaderboard, 8 policies, Pearson r = 0.989, ~100 H100 GPU-hours, seeded with real benchmark observations. 4. Runway, general world model policy evaluation (Feb 2026): r = 0.95 across eight policies. 5. U.S. FAA, 14 CFR Part 60: flight simulation training devices qualified and periodically re-evaluated against objective flight-test data. 6. Press coverage, 2nd World Humanoid Robot Games, Beijing, Aug 2026 (AP, CBS, Newsweek): 9.39 s 100 m; falls, fires, failure-to-stop incidents. 7. Physical Intelligence, "Precise Manipulation with Efficient Online RL" (RL Tokens), March 19, 2026: ~15 minutes of robot data within a two-hour training run, balance consumed by resets and overhead. 8. Georgia Tech Robotarium: 16,500+ remote experiments; facility automation at ~500 experiments/month; NSF/ONR-funded free academic access. 9. Goldman Sachs humanoid shipment forecasts (2035 horizon); public OEM platform announcements. 10. C. Finn, Y Combinator AI Startup School keynote (Aug 2026): reliability without babysitting as the frontier; phase framing per Physical Intelligence founders' public remarks. 11. π0 platform disclosures (Physical Intelligence, 2024–25): UR5e, Franka, bimanual Trossen/ARX/AgileX, mobile ALOHA-class bases. 12. MarketsandMarkets / Global Market Insights (2026): global TIC market ~$254–266B; Bureau Veritas outsourcing-share estimate. 13. MarketsandMarkets (2026): CRO services ~$85.4B (2025) against ~$300B global pharma R&D. ## 10. The white paper The full treatment — interval choice, auto-sizing, paired designs, cluster taxonomies, the three-layer market model, and the sim-agreement methodology — is in the Roborama white paper (40 pp., 44 sources), available on request to teams evaluating the platform. [Request the paper](mailto:research@roborama.com?subject=White%20paper%20request) and we'll send the current draft.