# End-to-End Imitation Learning for SO-101 with ROS 2
**URL:** https://discourse.openrobotics.org/t/end-to-end-imitation-learning-for-so-101-with-ros-2/53046
**Category:** Projects
**Tags:** ros2, rviz, jazzy
**Created:** [March 9, 2026, 3:14am UTC](https://discourse.openrobotics.org/t/end-to-end-imitation-learning-for-so-101-with-ros-2/53046 "2026-03-09T03:14:29Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@dmitri\_manajev](https://discourse.openrobotics.org/u/dmitri_manajev)
#### Post date: [March 9, 2026, 3:14am UTC](https://discourse.openrobotics.org/t/end-to-end-imitation-learning-for-so-101-with-ros-2/53046/1 "2026-03-09T03:14:29Z")
Hi everyone,
I’d like to share **SO-101 ROS Physical AI** , an open-source ROS 2 stack for the SO-101 robot arm focused on an end-to-end imitation learning workflow on real hardware.
GitHub repo: [legalaspro/so101-ros-physical-ai](https://github.com/legalaspro/so101-ros-physical-ai)
The idea behind this project was to build a **ROS 2-native full pipeline** around a low-cost, accessible robot arm that people can actually learn on in practice.
I wanted something that could be useful for **students, researchers, and hobbyists** , while also serving as a practical reference for implementing similar ROS 2 concepts on other robot arms.
That full workflow was the main motivation for the project: **not just isolated demos, but a ROS 2-native path from data collection to execution**.
## **What the repo includes**
- **Real robot bringup** for the SO-101 arm
- `ros2_control` **integration** with a Feetech STS3215 hardware interface
- **Leader/follower teleoperation** for collecting demonstrations
- **Multi-camera support**
- **Episode recording** for imitation learning data collection
- **ROS-to-Rerun visualization** for observations, actions, camera streams, and reviewing MCAP files
- **Rosbag / MCAP to LeRobot dataset conversion**
- **Policy inference**
- sync on-device inference
- async remote inference through a policy server for **action-chunking LeRobot policies** such as **ACT** , **SmolVLA** , **π0/π0.5** , and more
## **Current inference support**
At the moment, the repo includes:
- on-device sync inference for **ACT** and **SmolVLA**
- async remote inference for **action-chunking LeRobot policies** via `policy_server`
One of the parts I especially wanted to make practical was remote inference: the robot runs the ROS 2 client locally, while the policy can run on a remote GPU server.
## **Typical workflow**
1. Teleoperate the robot in a leader/follower setup
2. Record demonstration episodes as rosbags / MCAP files
3. Convert recorded episodes into a LeRobot dataset
4. Train a policy with LeRobot
5. Run the learned policy back through ROS 2 on the real robot
## **Why I built it**
There are already many strong tools in both the ROS 2 and robot learning ecosystems, but I could not find a single **reusable ROS 2 example** covering the full imitation learning workflow for manipulation on real hardware:
- real robot bringup
- teleop for demonstrations
- dataset collection
- conversion into LeRobot format
- local or remote inference from ROS 2
For people who want to learn **learning-based control in robotics** , especially **imitation learning** , this felt like an important missing piece.
I also wanted it to be a practical platform that people could learn from and build on in practice.
## **Demo highlights**
I also recorded a few short videos showing different parts of the pipeline:
- **Async policy inference with a remote GPU server** [Watch on YouTube](https://www.youtube.com/watch?v=l6kWDoHxczc)
- **Episode recording workflow** [Watch on YouTube](https://www.youtube.com/watch?v=lXkloZll4PA)
- **Teleoperation for SO-101 arms in ROS 2** [Watch on YouTube](https://www.youtube.com/watch?v=wLBEH63x_nA)
- **Final LeRobot dataset after rosbag conversion** [Watch on YouTube](https://www.youtube.com/watch?v=ffXxCkYJ6as) , [Dataset page on Hugging Face](https://huggingface.co/datasets?other=so101-ros-physical-ai)
- **SO-101 ROS 2 policy inference demo** [Watch on YouTube](https://www.youtube.com/watch?v=fpGmTwjTmzM)
## **Feedback welcome**
I’m still actively improving the stack, and I’d love feedback from people working at the intersection of:
- ROS 2
- `ros2_control`
- LeRobot
- imitation learning
- remote policy inference
I hope this can be useful both as a practical project and as a learning resource for people exploring imitation learning on real robots with ROS 2.
Thanks!
At the end, I also included a short demo showing a simple way to run async inference in ROS 2 with a remote policy server:
[ ](https://www.youtube.com/watch?v=l6kWDoHxczc)
---
### Author:  [@Katherine\_Scott](https://discourse.openrobotics.org/u/Katherine_Scott)
#### Post date: [March 9, 2026, 6:51pm UTC](https://discourse.openrobotics.org/t/end-to-end-imitation-learning-for-so-101-with-ros-2/53046/2 "2026-03-09T18:51:01Z")
Wow, this is awesome. I think you should reach out to our Physical AI working group to present this project.
---
### Author:  [@dmitri\_manajev](https://discourse.openrobotics.org/u/dmitri_manajev)
#### Post date: [March 9, 2026, 8:11pm UTC](https://discourse.openrobotics.org/t/end-to-end-imitation-learning-for-so-101-with-ros-2/53046/3 "2026-03-09T20:11:22Z")
Thanks you so much, I really appreciate that. I’d be happy to show it to the Physical AI working group. I’m just not sure, who I should reach out to. 😅
---
### Author:  [@Alberto\_Ezquerro](https://discourse.openrobotics.org/u/Alberto_Ezquerro)
#### Post date: [March 10, 2026, 11:41am UTC](https://discourse.openrobotics.org/t/end-to-end-imitation-learning-for-so-101-with-ros-2/53046/4 "2026-03-10T11:41:13Z")
Awesome work @dmitri_manajev ! This really looks like an amazing project to get started with Imitation Learning and Physical AI with ROS 2.
---
### Author:  [@dmitri\_manajev](https://discourse.openrobotics.org/u/dmitri_manajev)
#### Post date: [March 10, 2026, 8:32pm UTC](https://discourse.openrobotics.org/t/end-to-end-imitation-learning-for-so-101-with-ros-2/53046/5 "2026-03-10T20:32:35Z")
Thank you so much, Alberto — I really appreciate it. I’m also very grateful to The Construct Robotics Institute, since this project originally started as my final Masterclass project and a way to showcase what I learned during the course. 🦾