I’ve been exploring how AI agents are being developed and used to interact with robotic systems, especially within ROS and ROS 2 environments. There are some exciting projects in the community — from NASA JPL’s open‑source ROSA agent that lets you query and command ROS systems with natural language, to community efforts building AI agents for TurtleSim and TurtleBot3 using LangChain and other agent frameworks.
I’d love to start a discussion around AI agent design, implementation, and real‑world use in robotics:
Which AI agent frameworks have you experimented with for robotics?
For example, have you used ROSA, RAI, LangChain‑based agents, or custom solutions? What worked well and what limitations did you encounter?
How do you handle multi‑modal inputs and outputs in your agents?
(e.g., combining natural language, sensor data, and robot commands)
What strategies do you use for planning and action execution with your agent?
Do you integrate RL policies, behavior trees, skill libraries, or other reasoning approaches?
What tooling or libraries do you recommend for scalable agent performance?
Have you found certain profiling tools, API integrations, or frameworks particularly helpful?
What are the biggest challenges you’ve faced when deploying your AI agent on real robots?
(e.g., latency, safety, unexpected robot behavior, or integration issues)
Are there any resources, examples, or papers that helped you with agent development?
I’m keen to share references and compare experiences.
Let’s share our experiences and recommendations — whether you’re just starting to explore AI agents or you’ve already built something that interacts with real robotic systems!
I only now noticed this post. Our team both builds RAI and builds on RAI (e.g. specific projects). Let me know if you are still interested in answers - one of good channels to explore is the Embodied AI Community Group Discord, ROS Embodied AI Working Group . I’ll also ask my colleague to take a look and answer this for you.
A bit late, but this is squarely my area, so here’s my experience with the caveat that I come at it from an unusual angle.
I don’t build the agent; I build the layer underneath it, a deterministic governance runtime called XZane. So my lessons are less “which framework proposes actions well” and more “what happens between the model’s proposed action and the hardware.”
On frameworks and models: it’s deliberately model and framework-agnostic, because the enforcement layer shouldn’t care who proposes the action. I’ve run it across five providers on real hardware (Claude, GPT-4.1, Gemini, and local Qwen/Mistral on a Jetson-class board) switchable at runtime, each output validated identically, with the tool surface exposed over MCP. One limitation worth flagging: providers vary a lot in how often they emit malformed or out-of-envelope commands, so “works with model X” doesn’t transfer cleanly without re-checking.
On planning and execution: I don’t impose a paradigm, no RL or behaviour trees baked in. The model plans; the runtime validates. Proposed actions are checked against a declared capability profile (permitted actions, parameter bounds, operational envelope) before dispatch; sequences are checked for temporal and dependency sanity; and complex tasks can be delegated to pre-validated modules that run semi-autonomously and report back. Honest boundary: I govern commands and consume perception state, but I don’t do the 2D→3D world-state lifting a proper VLA harness does… different layer.
The real-hardware challenges are where I’ve learned the most:
Risk is invisible at the text layer. My clearest result: a prompt-injection attack fooled all five providers, but the physical action was still blocked, because the block comes from an independent layer firing on telemetry (LiDAR/IMU), not from reading the model’s output. If safety depends on inspecting the agent’s text, in my opinion it’s watching the wrong place.
Transient sensor blind spots. A momentary LiDAR dropout mustn’t read as “path clear,” so a short temporal memory on detections matters more than you’d expect.
Worth saying clearly: this is complementary to the agent frameworks in this thread, not a competitor to them: RAI, ROSA and others propose and orchestrate; the runtime just decides whether a proposed action reaches the actuators. Happy to go deeper on any of it, and genuinely curious how others here are drawing that propose-versus-govern line, especially anyone who’s taken an agent past simulation.