Embodied-sync: Synchronize, calibrate, align, and validate multimodal robot-learning data across sensors, clocks, live sessions, and recordings

Often, robot-learning sensor streams or datasets need to be synchronized. Cameras run at one rate, robot state at another, packets arrive late, and a device reconnect can silently reset its time offset. When you are trying to finish an experiment, the last thing you need is to discover after training that the observations were paired differently on the robot than they were in the dataset.

embodied-sync gives you one place to align, replay, inspect, and validate multimodal timing. It works with both live sensor streams and recordings, and it fits around the tools you already use: UMI, LeRobot, ROS 2/rosbag2 + MCAP, LSL/XDF, Rerun, and SurgSync-style datasets. It also attempts to adapt to any custom dataset format, with an interactive dialog allowing users to pick the right choices for alignment/format. It also generates reports with detailed information like downsampled frequencies, dropped frames and latency and provides the ability to inspect frames/audio to verify.

Current alpha users include QUT Robotics, hobby enthusiasts working with camera/IMU streams/tactile device streams, robotics startups. The methods have been validated on several real-world datasets including RocSync, SurgSync, QUT manipulation learning datasets and ROSBag2.

2 Likes

The validation half is the interesting half, and it is upstream of a problem I hit constantly.

Downstream of you, I run matched-pair evaluations on robot policies, and the whole design falls apart if the two arms are not actually comparable. We shipped a result where the policy’s own sampler was unseeded, so the number was one draw rather than a reproducible constant. Caught it late, fixed it in the runner, and it cannot be applied retroactively to a measurement already taken.

So a question: does Embodied-Sync record enough provenance that a third party could re-run one
recorded session and get the same frames back? Clock alignment plus a seed record is the difference between a dataset you can compare against and one you can only look at.

Thanks, that’s good feedback! Created an issue to track this: Add deterministic replay provenance and verification · Issue #1 · anicut-ai/embodied-sync · GitHub

@Sattyam_Jain please take a look at this PR Add provenance and replay by bug-catcher · Pull Request #2 · anicut-ai/embodied-sync