End-to-End Imitation Learning for SO-101 with ROS 2

dmitri_manajev
Mon, 09 Mar 2026 03:14:29 +0000
Projects

Hi everyone,

I’d like to share SO-101 ROS Physical AI, an open-source ROS 2 stack for the SO-101 robot arm focused on an end-to-end imitation learning workflow on real hardware.

GitHub repo: legalaspro/so101-ros-physical-ai

The idea behind this project was to build a ROS 2-native full pipeline around a low-cost, accessible robot arm that people can actually learn on in practice.

I wanted something that could be useful for students, researchers, and hobbyists, while also serving as a practical reference for implementing similar ROS 2 concepts on other robot arms.

That full workflow was the main motivation for the project: not just isolated demos, but a ROS 2-native path from data collection to execution.

What the repo includes

Real robot bringup for the SO-101 arm

ros2_control integration with a Feetech STS3215 hardware interface

Leader/follower teleoperation for collecting demonstrations

Multi-camera support

Episode recording for imitation learning data collection

ROS-to-Rerun visualization for observations, actions, camera streams, and reviewing MCAP files

Rosbag / MCAP to LeRobot dataset conversion

Policy inference

sync on-device inference

async remote inference through a policy server for action-chunking LeRobot policies such as ACT, SmolVLA, π0/π0.5, and more

Current inference support

At the moment, the repo includes:

on-device sync inference for ACT and SmolVLA

async remote inference for action-chunking LeRobot policies via policy_server

One of the parts I especially wanted to make practical was remote inference: the robot runs the ROS 2 client locally, while the policy can run on a remote GPU server.

Typical workflow

1. Teleoperate the robot in a leader/follower setup

2. Record demonstration episodes as rosbags / MCAP files

3. Convert recorded episodes into a LeRobot dataset

4. Train a policy with LeRobot

5. Run the learned policy back through ROS 2 on the real robot

Why I built it

There are already many strong tools in both the ROS 2 and robot learning ecosystems, but I could not find a single reusable ROS 2 example covering the full imitation learning workflow for manipulation on real hardware:

real robot bringup

teleop for demonstrations

dataset collection

conversion into LeRobot format

local or remote inference from ROS 2

For people who want to learn learning-based control in robotics, especially imitation learning, this felt like an important missing piece.

I also wanted it to be a practical platform that people could learn from and build on in practice.

Demo highlights

I also recorded a few short videos showing different parts of the pipeline:

Async policy inference with a remote GPU server Watch on YouTube

Episode recording workflow Watch on YouTube

Teleoperation for SO-101 arms in ROS 2 Watch on YouTube

Final LeRobot dataset after rosbag conversion Watch on YouTube , Dataset page on Hugging Face

SO-101 ROS 2 policy inference demo Watch on YouTube

Feedback welcome

I’m still actively improving the stack, and I’d love feedback from people working at the intersection of:

ROS 2

ros2_control

LeRobot

imitation learning

remote policy inference

I hope this can be useful both as a practical project and as a learning resource for people exploring imitation learning on real robots with ROS 2.

Thanks!

At the end, I also included a short demo showing a simple way to run async inference in ROS 2 with a remote policy server:

Async ROS 2 Policy Inference with a Remote Server on Vast.ai

5 posts - 3 participants