rhoyn

Stable walk

github.com/rhoyn/teleop-walking-benchmark

If you need high-quality humanoid teleoperation datasets, contact us hello@rhoyn.com

This measures walking, it does not solve it — the policies are other people’s work. Low latency eyes and low latency, accurate hands measured the link to the robot; this is the far end: you can have a 20 ms connection to a robot that falls over.

Thirty-four policies and no way to choose

Open-source G1 controllers arrive steadily, each with a checkpoint and a demo video where it works. None are comparable, so we ported thirty-four to one C++ interface and ran every one of them against the same task in two different physics engines.

The task

Ninety seconds, eighteen waypoints, 50 Hz control over a 2 ms step. A crane holds every candidate in the same stance for three seconds and then lets go; the clock starts there. Each segment lands one punch — random joint, random direction, 0.08 s, up to 500 N, with the ceiling ramping from a third of that at the start to the full 500 N by the sixtieth second — drawn from the run id before the run starts, so it cannot be dodged or outrun. The score is recovery, not tracking.

The arms are not the policy’s to use. A policy owns the fifteen leg and waist joints. The fourteen arm joints random-walk under the harness — never still, never borrowable for balance — because a controller that windmills to stay upright falls the moment its operator reaches for a door handle.

One run id, mid-tour, every ranked row in both engines: forty-two policy rows, each stepped once by MuJoCo and once by PhysX, eighty-four tiles.

The whole run, those eighty-four side by side:

The campaign

144,384 runs, both engines, in rounds of 512 run ids. A run id names one whole task — its own tour and its own punches — so a run id is a task rather than a repetition. A policy earns depth by tier: tier A, the eleven rows that complete at least 60% averaged over the two engines, ran ten rounds, 5120 run ids; tier B ran one, since a policy that falls most of the time needs no fourth decimal place. The runs column below carries each row’s own count, so no row is read against a total it never had. Every policy and every engine sits behind the same CUDA interface, which is what makes a campaign this size a benchmark rather than a demo.

If you need high-quality humanoid teleoperation datasets, contact us hello@rhoyn.com

Results

Completions first, because every other column is read against them. Rows are in the README’s order, by mean survival over both engines. The upper bar is MuJoCo, the lower PhysX; the names in bold are the ones you can drive yourself further down the page.

Every cell is MuJoCo / PhysX. Survival is mean seconds upright, the column the table is ranked on. Errors are means over targets reached. The last two columns are what a completed walk cost, and an engine’s cell is blank when fewer than a fifth of its runs finished.

The last two rows are greyed out because they are unranked: each is a policy handed all twenty-nine joints instead of fifteen. handoff goes from 0/0% to 34/44% that way and clobot from nothing to 2/5%, which is the cost of the arms rule stated as a number. The four names marked * read the base linear velocity straight from the simulator, a sensor no real robot has; they are ranked, but not deployable as shipped.

Reading it

The top eight rows are two policies under four postures each. decoupled_wbc’s authors, after reviewing our adapter, asked us to sweep the commanded torso height and pitch; we then ran gr00t_wbc over the same four. Commanded height is the only parameter that moves the score — pitch does nothing for completion in either policy, and a wider sweep over gr00t_wbc’s walk/balance switch found it inert.

Lowering the torso buys completion, and gr00t_wbc pays for it in accuracy. Its canonical 0.74 m posture takes 63/67% at 13/15 cm of position error; 0.70 takes 73/75% at 14/19 cm; 0.66 takes 80/72% at 17/51 cm. In MuJoCo that is seventeen extra points of completion for four centimetres; in PhysX the last step buys nothing and leaves the robot more than three times as far from where it was sent. decoupled_wbc pays far less: across all four postures its position error holds at 10–11 cm in MuJoCo, the lowest in the table, while completion climbs from 69/62% to 75/68%.

Both beat their own authors’ defaults. Neither canonical posture is that policy’s best row: gr00t_wbc’s 0.74 m row is the worst of its four, and decoupled_wbc’s 0.70 m row trails both of its 0.66 m rows. Read the completion column against the error column, or the top of this table will mislead you.

Completing is not the same as going where you were sent. homie finishes 55/58% of its tours with 32/36° of yaw error — it arrives, but it does not arrive facing the right way. grove sits a few centimetres behind homie, 21/22 cm against 17/18, but holds heading to 7/6°, and pays for it: 12.4/13.2 kJ and 2.1/2.4 vibration units a walk, the most expensive gait in tier A.

Low error does not mean good. wty_cpp posts 31/32 cm, within a few centimetres of amo and wbc_agile, and completes 0/4%. falcon posts 42/49 cm and completes 2/4%. Errors are averaged only over targets actually reached, so falling early flatters a candidate. Read them against the completion column or not at all.

Nothing completes reliably. The best row in the field still fails about a quarter of its tours. Below amo nothing clears two in five, and the last twenty complete essentially nothing in either engine — twenty open-source G1 controllers that cannot walk a punched tour at all.

MuJoCo against PhysX

Every row is one policy run twice: the same exported weights, the same waypoints, the same punches, stepped by two different physics engines. Nothing about the policy changes between the two halves of the completion column. Only the contact model does.

They disagree because contact is solved differently. MuJoCo integrates soft convex constraints; PhysX runs a TGS solver over rigid contact patches; neither is the real robot. Both step on the GPU, MuJoCo through MuJoCo Warp and PhysX through its own GPU solver. A policy that scores in one engine has learned that engine’s contacts. One that scores in both has survived a change of contact model it never saw in training, which is the closest thing here to a sim-to-real proxy.

The gaps are not noise. sonic takes 36% in MuJoCo and 15% in PhysX — the same network, well under half the result. grove runs the other way, 49% against 63%, and mimic_lite 9% against 23%. Read the two numbers together, or you will ship the wrong policy.

One more thing worth knowing: neither engine repeats a run id bit for bit. Both step on the GPU and reduce contact forces across threads in arbitrary order, so the same run id can pass one round and fall the next. That is why tier A ran ten rounds rather than one long one: compare aggregates across rounds, not individual run ids.

What not to read into it

This is simulation, not a floor. It is the right place to narrow thirty-four candidates to a shortlist, and the wrong place to end the argument. The cost columns cover only the rows that finish often enough to average — the top twelve in both engines, sonic and mimic_lite in one; for the rest the completion column is the whole story.

The licences matter more than you would like

Of the thirty-four, two are non-commercial, eleven declare no licence at all, and one is sim-only at its authors’ request. homie is CC BY-NC, so a commercial build must exclude it. The permissive one is close behind the top: decoupled_wbc is BSD-3-Clause, best on position error, cheapest walk in MuJoCo, and within a few points of gr00t_wbc on completion. gr00t_wbc, on top, is under the NVIDIA Open Model licence. Terms for every checkpoint are in the NOTICE.

Walk it yourself

Twenty-three of them run in this page: pick one and it downloads that policy’s real weights, then runs the real network at 50 Hz against real MuJoCo physics in your browser. Each is a port of the same policy.cpp the benchmark runs — the same observation layout, gait clocks, gains, offsets and action clamps — so what you drive is what was scored. gr00t_wbc is its 0.66 m posture and decoupled_wbc its 0.66 m / 0.12 rad one, within a point of the best row of each.

Twenty-three and not thirty-four, because the rest cannot reach a browser. sonic is 1.6 GB across three networks and bfm_zero 123 MB, against a 20 MB budget; mimic_lite runs on sonic’s planner, so it goes with it. robomimic, g1_gym and stepdown ship only TorchScript, which a browser cannot run. wcompton has no redistributable weights at all. huru, josabb, sunny and mturan33 read the simulator’s base velocity, which the page does not supply.

Most of them fall over, and that is the honest result rather than a broken page. Only the top six complete tours in the benchmark; the rest are here so you can watch what 0% actually looks like.

Starting on frozen — the stance the crane holds, with no policy running. Pick one above and it downloads that policy’s weights, a few MB on top of the physics engine.

Click it, then drive: W/S forward and back, A/D sideways, Q/E turn. Nothing is punching it here, so whatever goes wrong is the policy on its own. Each one is capped at the same command range the benchmark gives it, which is why amo will not turn on the spot and wbc_agile will outrun the rest.

Adding a policy

One directory and two lines: implement init, step and name with the weights beside them, then add the include and a branch in the factory. Everything is open sourced at github.com/rhoyn/teleop-walking-benchmark, MIT where it is ours to license. Read the NOTICE first.

If you need high-quality humanoid teleoperation datasets, contact us hello@rhoyn.com

Check our other blog posts

rhoyn is a humanoid data company. We build bleeding-edge teleoperation software and use it to collect high-quality datasets in diverse environments, then provide them to the teams training humanoid robots.