We recently conducted an experiment connecting GPT-6 Astra and AI Worker through Codex so the AI can perceive and control a physical robot, and we have published the full Technical Story and demo video.
Without task-specific policy training, we wanted to verify how far a general-purpose model can go when it selects and combines available robot motion and control tools based on context, and determines its next actions from real execution results.
The experiment focuses on three core areas:
- Situation Assessment: Environment perception based on camera images and robot state
- Task Planning: Multi-step planning and execution from natural-language instructions
- Robot Control: Feedback-driven object manipulation and mobile manipulation
Here is the system architecture and execution flow:
- Model Inference and Tool Calling: Connected via the Codex App Server, GPT-6 Astra checks the current camera view and robot state, then calls functions that specify the absolute target position and orientation for the selected hand.
- Harness and ROS 2 Control: The harness validates the model’s tool calls and translates them into ROS 2 and control commands to drive the robot.
- Observation and Feedback Loop: Once a motion finishes, the harness collects updated camera images, robot state, and execution results to return to the model, which then determines the next action based on that feedback.
In the documentation, we also organized the reproducible execution environment and the engineering insights gained during testing:
- Simulation and Real Hardware Support: You can run the exact same structure in both the Gazebo tabletop environment and on the physical AI Worker.
- Isolated Credential Path: We configured the environment variables so the harness references authentication credentials without conflicting with existing setups.
- Real-Time Monitoring: Through the video bridge and viewer, you can track live camera streams and per-episode execution logs directly in your browser.
- Safety Parameters and Tradeoff Analysis: We cover how to configure the maximum step distance, maximum action count, and time limits, and openly share the pros and cons observed during testing, including visual feedback recovery cases alongside limitations in 2D image depth estimation, collision conditions, repeatability, and response latency.
Although current limitations exist in API latency and execution reliability, this experiment confirms the potential of Agentic AI × Physical AI by connecting AI reasoning directly to physical robot actions. You can check out the full setup instructions and execution procedure in the documentation and video below.
Video: https://youtu.be/ZQXV_ngVG20
Docs: GPT-6 Astra on the AI Worker | ROBOTIS Docs
Community: ROBOTIS | Actuator for Physical AI