Smart Pick-and-Place on OMY with GraspNet and Gemini

Projects
1

An end-to-end smart pick-and-place pipeline built on the OMY robotic arm, integrating Gemini and GraspNet.

Here is how the pipeline works:

  1. Scene Reasoning: Gemini interprets natural language instructions and identifies target objects from camera inputs.
  2. Grasp Generation: GraspNet processes 3D point cloud data to generate stable 6-DoF grasp candidates.
  3. Motion Execution: Cyclo Control, our QP-based motion controller, plans collision-free trajectories and executes inverse kinematics in real time.

By combining VLM reasoning, data-driven grasp pose synthesis, and constrained motion control in ROS 2, OMY reliably picks arbitrary objects even in unstructured setups.

Check out the full workflow, demo video, and open-source code below.

:clapper_board: Video: https://youtu.be/OxMBVGkFJHM

:robot: Code:
• GitHub - ROBOTIS-GIT/cyclo_control: Control packages for the ROBOTIS Physical AI lineup · GitHub
• GitHub - ROBOTIS-GIT/open_manipulator: AI Manipulator and Open Manipulator · GitHub

:memo: Docs: Pick-and-Place on OMY with GraspNet and Gemini | ROBOTIS Docs