How long does a humanoid take to reproduce a movement of the operator’s hand? Video is limited by its frame rate and software logs stop before the motors, so we put one IMU on each hand and read both on one clock at 1 kHz. Ten estimators from six method families measure the lag and we report their median; known delays injected into the recordings come back within 0.7 ms. Across 21 recordings on a Unitree G1 EDU+, rhoyn’s VR teleop stack follows the operator 2.2× and 4.4× faster than two other stacks, with no overlap between the three, and its takes vary least.
What you can reuse
- The rig: two MPU6050s and one ESP32, firmware included. It times hand to hand at 1 kHz, wherever the boards are taped.
- The analysis: one command turns a recording into a latency and a picture of how all ten estimators got there.
- The data: all 21 recordings and a video of every take, MIT licensed like the code, plus each take’s report grid. github.com/rhoyn/teleop-hands-latency-analysis
1Why 20 ms vs 90 ms shows up in your data
A person steering anything runs a closed loop: look, correct, look again. Put a delay in it and you stop correcting toward where the target is and start correcting toward where it was — the textbook recipe for overshoot. You push, nothing happens, you push harder, and then everything you asked for arrives at once. At 20 ms you don’t notice. At 90 ms you are placing the gripper by trial and error.
In teleoperation those corrections can be recorded into demonstrations — the overshoot, the re-tries, the pauses spent waiting for the robot — and a policy trained on them may learn them as part of the task. The delay also sits between the recorded command and what the camera sees: at 90 ms that is almost three frames of a 30 Hz camera, and when the delay changes from take to take, no single time shift at training time lines them up again.
2Video and software timestamps can’t resolve it
Counting frames. The ExtremControl paper estimates the latency of ten humanoid teleoperation systems, its own among them, by optical flow on their released videos — a clever way to get numbers for systems you don’t have in the room. But the result is limited by each video’s frame rate: one frame at 30 fps lasts ~33 ms, longer than rhoyn’s whole latency.
One of our takes at 60 fps: the operator’s hand enters from the left, the robot’s gloved hand from the right. At 20 ms the eye can’t tell which one is behind.
Timestamping software. It stops at the last line of code, before the motor and the limb. And it subtracts timestamps taken on two machines, so the answer includes however wrong their clock synchronisation is.
3Two hands, one clock
Two MPU6050s, one on each hand, are read by the same ESP32 at 1 kHz, and each pair of samples is stamped with one micros() call. There is no clock offset between devices to estimate, so the lag we measure is physical delay between the hands.
Nothing in the instrument adds lag to one hand only. Both hands travel in one packet, and both get identical filtering. Both sensors are read in parallel on two I2C buses and sent in the same 31-byte packet. A lost packet loses both hands together, and a failed read can spoil a sample but never shift it, because the packet still goes out on the same tick, so the two hands can never slip apart in time. Both sensors use the same on-chip low-pass setting, and every estimator high-passes both signals at 0.3 Hz; any smoothing an estimator adds is the same on both hands, so filter delay cancels in the difference.Why the instrument is fair to both hands
Mounting doesn’t matter. Every estimator works on gyroscope magnitude |ω|, which is the same however a board sits on its hand, so the boards can be taped on crooked and gravity never enters the signal.
The sensors clip on fast reaches. They run at their default ±250 dps, and rhoyn’s takes clip most. Re-clipping every recording harder moves rhoyn by 0.3–1.7 ms and leaves the ranking unchanged. All 21 recordings hit full scale. On the robot hand, rhoyn’s takes clip on 18–51% of samples, ExtremControl’s on 7–11% and GR00T’s on 1–14%. A clipped peak is flat, so it carries less timing information. We re-ran the analysis on copies of all 21 recordings clipped harder still: Clipping both hands harder raises all three medians, the slower stacks most; clipping one hand only can move a take either way. The ranking and the gap between the stacks hold in every case. (At 100 dps one rhoyn take has 9 of 10 estimators.) This test can’t show what an unclipped recording would read — only that the conclusions don’t hinge on clipping. A recording at ±1000 dps would remove the question.The clipping stress test
clipped at rhoyn XRoboToolkit + ExtremControl XRoboToolkit + GR00T ratios 250 dps (as recorded) 20.3 ms 44.5 ms 89.3 ms 2.2× / 4.4× 200 dps 21.0 ms 45.8 ms 91.7 ms 2.2× / 4.4× 150 dps 21.4 ms 46.8 ms 97.0 ms 2.2× / 4.5× 100 dps 22.0 ms 49.1 ms 102.6 ms 2.2× / 4.7× 150 dps, robot hand only 20.7 ms 45.8 ms 94.8 ms 2.2× / 4.6× 150 dps, operator hand only 20.0 ms 45.4 ms 91.3 ms 2.3× / 4.6×
4Ten estimators, one median
A single delay estimator can hand you a confident, wrong number, and nothing in its output tells you so. So ten run over every recording, from six method families — cross-correlation, frequency-domain phase, model fitting, time warping, template and peak matching, and a Bayesian posterior — and the reported latency is their median. One method going wrong on a take can’t move a median far.
Rhoyn run 5 shows it: peak_matching read 31.5 ms while the other nine sat between 19.6 and 25.8, and the median stayed at 20.8 ms.
Figure 3. Every run writes one report: the result on top, then one tile per estimator showing its own internals. Run 0, rhoyn stack: median 20.6 ms, all ten between 18.3 and 26.0, typical distance from the median (MAD) 0.9 ms.
On rhoyn most of the ten land within a millisecond or two of the median, with the odd outlier like run 5. On the slower stacks they spread wider — GR00T run 1 reads anywhere from 81.5 to 115.7 ms — and the median still lands at 94.9 ms, inside the stack’s other takes.
Known delays come back within 0.7 ms. To test the whole chain, we fed it recordings with a delay we set ourselves:
| injected into all 21 recordings | read back |
|---|---|
| robot hand replaced by the operator’s, delayed 0 ms | 0.0 ms on every recording |
| … delayed 20 ms | 20.0 ms on every recording |
| … delayed 90 ms | 89.8–90.0 ms |
| real robot hand, delayed a further 20 ms | each result rises by 19.4–20.5 ms |
| real robot hand, delayed a further 50 ms | each result rises by 49.4–50.7 ms |
A delayed copy is easier than a real robot hand, which also shrinks and smooths the motion; the last two rows keep that reshaping and still recover the injected delay within 0.7 ms. All ten estimators reported in every case. peak_matching — 18.4 ms. The one to show someone who doesn’t want the signal processing: find the moment each hand moves fastest in every reach and measure the gap. The middle panel is one reach, two peaks, 20 ms apart. group_delay — 18.3 ms. A pure delay shows up as phase falling linearly with frequency; the slope is the answer. It reads phase only, so it isn’t biased when the robot shrinks your motion. kalman_bank — 19.6 ms. 201 Kalman filters, one per delay hypothesis from 0 to 200 ms, each scoring how well its assumption explains what arrived. It thrashes for half a second, then locks on. An estimator may abstain on a take it can’t measure; none did on any of the 21 recordings. Search windows are bounded — eight of the ten look at no more than 200–250 ms of lag — so a repetitive motion can’t easily be matched a whole cycle late. Samples sit on a 1 ms grid; most estimators refine their peak below that, while the two time-warping methods report whole milliseconds. The two model-fitting estimators (arx_delay, kalman_bank) share one model and preprocessing and agree closely, as do the two cross-correlation methods; the two time-warping methods share a method and usually agree. That is why we count six families, not ten independent opinions.Three estimators up close · peak matching, group delay, Kalman bank
All ten estimators · method of each
algorithm approach windowed_gcc windowed cross-correlation with rest gating and Gaussian peak refinement cross_correlation plain-weighted generalized cross-correlation, one tau per window group_delay Welch H1 transfer function, delay from the coherence-weighted phase slope arx_delay ARX model with an explicit delay parameter, searched over a grid kalman_bank 201 Kalman filters, one per delay hypothesis, likelihood vote dtw_offset banded dynamic time warping, offset of the warping path burst_dtw subsequence DTW per motion burst, median across bursts matched_filter one operator burst as a template, slid over the robot burst peak_matching the speed peak of each reach on both hands, and the gap between them bayesian_posterior posterior over tau with the amplitude gain profiled out, MAP plus 95% CI
5Results: the three stacks never overlap
Three stacks on the same Unitree G1 EDU+ in the same room: 9 recordings of rhoyn and 6 of each other stack, same two IMUs, same ten estimators. rhoyn’s slowest recording (21.6 ms) is faster than the fastest recording from either other stack (37.2 ms).
The spread matters as much as the median. rhoyn’s takes stay within 17.2–21.6 ms and ExtremControl’s range over 37.2–48.5 ms; GR00T’s varies over 69.5–98.7 ms, a spread of nearly 30 ms between takes. Full names: XRoboToolkit PICO VR + Genesis ExtremControl; XRoboToolkit PICO VR + NVIDIA GR00T low-latency weights.Show as a table
# rhoyn XRoboToolkit + ExtremControl XRoboToolkit + GR00T 0 20.6 ms 48.5 ms 88.2 ms 1 19.4 ms 47.7 ms 94.9 ms 2 17.2 ms 45.9 ms 69.5 ms 3 20.3 ms 43.2 ms 90.5 ms 4 17.7 ms 39.1 ms 87.7 ms 5 20.8 ms 37.2 ms 98.7 ms 6 21.6 ms — — 7 19.2 ms — — 8 20.7 ms — — median 20.3 ms 44.5 ms 89.3 ms
6Against the published field: a ranking, not a like-for-like
The ExtremControl paper ranks ten published systems, its own included, by optical flow on their videos. Every row there is bound by the video’s frame rate — at 30 fps, ~33 ms, more than rhoyn’s entire latency — so the chart below is an order-of-magnitude ranking, not twelve numbers measured the same way.
One system appears in both instruments, and that is the useful check. We measured XRoboToolkit + ExtremControl at 44.5 ms; the paper reports ~54 (MoCap) and ~64 ms (VR) for ExtremControl. Our rig reads it faster. An instrument biased against competitors would have read it slower.
7Limitations
- End to end: a compliant joint counts as much as a network hop, so it can’t tell you which link to fix.
- The robot shrinks and smooths your motion, which biases methods that expect a clean delayed copy; group_delay reads phase only and isn’t biased.
- It reports a magnitude, never a negative lag: the robot is assumed to follow.
- This is the instrument. How rhoyn reaches 20 ms is a separate article.
8Measure your own stack
Everything above runs from the repository. Two MPU6050 breakouts and one ESP32 are the whole rig: tape one board to the operator’s controller and one to the robot’s hand, put each on its own I2C bus (both boards answer to the same address), and flash esp32_firmware.cpp.
| wiring | bus A on pins 32/33, bus B on pins 25/26, both at 400 kHz; USB to the laptop at 921600 baud |
|---|---|
| record | ./record.sh --port /dev/ttyUSB0 --out inputs/my_take.csv — a healthy rig shows ~1000 samples/s with bad at 0 |
| analyse | ./run.sh inputs/my_take.csv outputs/my_take.jpg — prints all ten estimates and the median, and writes the report grid |
| needs | a C++17 compiler, Eigen and the stb headers |
Drive the robot in bursts, ten to twenty reaches per take, and run the same protocol on every stack you compare.
9Try it yourself
The simulation the argument in section 1 is about, with the control link starting at 90 ms. Click it, then T/G moves the control latency (this article) and R/F the camera latency (the eyes article). Set control to 20 ms, then 90 ms, and try to place a hand exactly where you meant to.