FULL TUTORIAL: Isaac SIM -> isaac_ros_foundationpose -> ManyMove

[pastoriomarco]

Hi everyone!

I just published the full isaac_ros_foundationpose pipeline tutorial with custom fine-tuned YOLOV8 model.

Here you find the YOUTUBE VIDEO!

KEY FEATURES

The pipeline includes:

• NVIDIA Isaac SIM 5.0 to generate synthetic data for fine-tuning the YoloV8s model and for a digital twin of the scene to publish a virtual RealSense camera stream and robot data to ROS2
• Ultralytics to fine-tune the YoloV8s model
• isaac_ros_foundationpose to estimate the 6D pose of the object, using custom fine-tuned YoloV8s for object detection
• ManyMove to handle the ROS2 logic and motion planning leveraging MoveIt2 and BehaviorTree.CPP

Hardware:

• NVIDIA Jetson Orin Agx Developer Kit for isaac_ros_foundationpose and ManyMove on ROS2 Humble
• Laptop with RTX card for Isaac SIM

HIGHLIGHTS

• All assets provided to complete the pipeline, from .usd files for SDG and scene to .obj and mesh for FoundationPose
• Example executable with ManyMove with behavior tree nodes to rectify FoundationPose output to allow grasping of symmetric objects and to limit pose validity to a specific bounding box: these features stabilize the output and enhance reliability on bin picking applications.
• Full Isaac Sim scene with Ufactory Lite6 cobot and gripper, customized to provide a realistic pneumatic gripper simulation while keeping coherent ROS2 interaction and MoveIt2 path planning.

LINKS

• ManyMove: GitHub - pastoriomarco/manymove: To plan and execute moves with manipulators in ROS2
• Isaac SIM: What Is Isaac Sim? — Isaac Sim Documentation
• isaac_ros_foundationpose: isaac_ros_foundationpose — Isaac ROS
• Ultralytics YOLOv8: GitHub - ultralytics/ultralytics: Ultralytics YOLO 🚀