Hi all, wanted to share a project I’ve been building and get some outside
eyes on it.
The problem I’m trying to solve
When you wire an LLM up to control a robot, the usual approach is a static
system prompt listing available actions. That list goes stale the moment
a node isn’t running, a service isn’t up, or the robot config changes.
The LLM ends up trying to call things that don’t exist, or that exist but
aren’t currently reachable.
What ros2_lingua does
Robots self-register their capabilities at runtime (name, description,
parameters, the actual ROS action/service to call). The LLM only ever
sees this live, grounded list, so it can’t invent a call that isn’t
actually there right now. A grounding/validation layer checks LLM output
against the registered schema before anything gets dispatched, and a
recovery layer handles retries and fallback behavior when a step fails.
Where it’s at
- Core Python package: capability registry, parameter validator, an
11-type error hierarchy, dispatcher node, and a recovery/replanning layer - C++ bindings (ros2_lingua_cpp) for nodes that aren’t in Python
- 80+ unit tests, 16 integration tests
- Multi-robot namespace support
- A small web dashboard over rosbridge for watching live capability
registration and dispatch status - LLM backends for OpenAI, Anthropic, and Ollama, Apache 2.0 licensed
It’s genuinely early. I’m one contributor, it hasn’t run on real hardware
outside a mock robot setup, and there’s no formal benchmarking yet on
hallucination rates or latency under load, that’s next.
Where I could use outside perspective
- Anyone who’s built LLM-to-ROS integrations before, does the
registration/grounding approach hold up against failure modes you’ve
hit in practice? - Contributors interested in a Gemini backend, C++ API parity work, or
CLI tooling, I’ve tagged several issues as good first issue with
specific line references if anyone wants to poke around - Real hardware testing beyond the mock robot setup would tell me a lot
I can’t learn from simulation alone
Repo: GitHub - purahan/ros2_lingua: Natural language to ROS2 actions — a structured LLM grounding engine for any robot. · GitHub (Apache 2.0)
Docs: ros2_lingua — Documentation
Happy to answer questions or walk through the architecture in more detail
if useful.