Good Afternoon Gazebo Community,
Over the past half year, I’ve been interested in integrating Reinforcement Learning (RL) within Gazebo in a manner that anybody can extend their RL problems into the Gazebo ecosystem, without having to worry about the fuss of Gazebo+Gymnasium. Since I know there’s been some interest in making this work, I began working on a package, gazebo_gymnasium. It’s an independent package installed via pixi which fuses the gz ecosystem with the all-so-popular Gymnasium RL library. It’s got the capability to extend into other Gazebo/ROS packages by using the bridge API, to make your own setup, removing the need to worry about Gazebo libraries.
To prove that this library works in a practical manner, we developed an end-end RL-trained line follower. We started by training in parallel in simulation, which it was then evaluated in simulation on various tracks
then using techniques like parallelization and domain randomization we again eval the more robust policy in simulation. Then actually deploy that on a real, custom differential drive rover that our lab team built, which worked (videos in the Google Drive) to follow various lines and tracks.
Before proceeding, please note that while the ideas, practices, and training was done by human much of the code was written with Claude, keep that in mind when using the repo.
There’s integration with popular RL libraries out of the box: Stable-Baselines3, skrl, rl_games, and rsl_rl. Along with Tensorboard, HuggingFace and Weights & Biases. We have some baselines testing and examples through the 3D Gymnasium examples which have policies you can test via Huggingface.
All in all, I know there’s still lots of work to do, more documentation to create, and there’s going to be problems when people go through it. But I’m hoping this is a good start to bridging the RL/Gz gap that currently exists in the modern community. I would very much enjoy questions, recommendations, and criticism if you can test/supply it.
Thanks, Lucas