Full pipeline Isaac SIM -> isaac_ros_foundationpose -> ManyMove

ROS ROS General
,
1

Hi everyone!
I’m sharing the progress on my project towards a complete bin-picking pipeline!

Here’s the link to the VIDEO.

I’ve enabled a complete pipeline featuring:

  • ManyMove to handle the ROS2 logic and motion planning leveraging **MoveIt2 **and BehaviorTree.CPP
  • Isaac SIM to create a digital twin of the scene and publishing a virtual RealSense camera stream and robot data to ROS2
  • isaac_ros_foundationpose to estimate the 6D pose of the object, using custom trained YoloV8 for object detection

In the video, the isaac_ros_foundationpose pipeline is running on a Jetson Orin AGX Developer Kit, while Isaac Sim and ManyMove/ROS2 Humble are running on a laptop with RTX GPU on Ubuntu 22.04.

Why ManyMove?

In the industrial applications I’m developing I need repeatability AND flexibility, and I need it to work on production now.
I’m also exploring the integration of VLA models to improve robustness and adaptability in the future. But, for certain scenarios, this kind of pipeline can get the job done reliably, and I have robots in production desperately needing a bin picking solution!

Feedback needed!

Little by little, I’m adding some tutorials and documentation to ManyMove. Some parts are still in heavy development, and the FoundationPose integration in particular is still quite green, and still specialized on this particular task.
I’d love if my work were of some use to other developers and roboticists, so any feedback is more than welcome!

A few areas where I’d especially appreciate your feedback:

  • Is it hard to understand how to use behavior trees in ManyMove? See this tutorial for example.
  • Would it be useful a tutorial on how to train a YoloV8 model on your custom objects to use in FoundationPose pipeline?
  • Would you be interested in a tutorial on how to integrate a pneumatic gripper that works with MoveIt2 and Isaac SIM? Yep, electric grippers are all the rage and for good reasons, but the cheaper pneumatic ones are great for some industrial applications that don’t need fine finger control, but just reliable grasping.