Organization: OSRF
Contributor: Ammaar Ahmed (GitHub, LinkedIn)
Mentors: Kimberly McGuire (GitHub) and Skyler Medeiros (GitHub)
GSoC project: ROS 2 Client Library Performance Monitoring
Repository: ros2-performance-monitoring
Live dashboard: performance.ammaar.lol
Hello everyone,
I am Ammaar Ahmed. I am pursuing a bachelor’s degree in Robotics and Automation Engineering at FAST NUCES Islamabad, Pakistan. My interest in ROS began about a year and a half ago, when I started building robots with ROS running on raspberry pi.
This summer I have been working on ROS 2 client library performance monitoring with OSRF, with guidance from Kimberly McGuire and Skyler Medeiros. The project is still in progress, but its main workflow now works from benchmark execution to a public dashboard. I wanted to share what it does, what has been completed, and what I plan to improve during the rest of the GSoC period.
Why this project exists
ROS 2 gives developers several choices for how an application communicates and runs. These include different client libraries, middleware implementations, executors, communication modes, and process layouts. These choices are useful because robots have different needs, but they also make performance difficult to compare.
A change that works well for small messages may behave differently with large messages. Results can also change when nodes move from one process to several processes. Looking at one number without knowing how it was produced can therefore give the wrong impression.
ROS 2 benchmark tools already produce detailed measurements, but their output is spread across many files and test scenarios. Reading those files by hand makes it difficult to answer common questions:
- Did this ROS 2 version become faster or slower?
- Is a slowdown widespread, or limited to one demanding test?
- Did latency improve at the cost of CPU or memory?
- Were both runs produced by the same workload?
This project connects benchmark execution, result processing, and visualization. Its purpose is to make the results easier to reproduce, compare, and understand.
Current Status and Workflow
The project provides a CLI workflow for running a reduced rclcpp benchmark matrix. It covers publish/subscribe communication, client/service calls, multiple message sizes, middleware implementations, communication modes, and both single-process and multi-process layouts.
The ros2-performance-monitoring run command executes the supported benchmark suite and converts the raw benchmark outputs into a consistent JSONL format. The dashboard up command launches local Prometheus and Grafana services for interactive analysis. Docker and the Docker Compose plugin are required. The project also supports container and image reuse, CPU pinning, alternative ROS distributions, separate Pub/Sub or service suites, and additional configuration options described in the README.md.
The workflow is:
Run a ROS 2 benchmark
↓
Collect benchmark results
↓
Normalize measurements
↓
Compare runs in the dashboard
The dashboard helps answer both what changed and why it changed . The default view compares two benchmark runs, summarizes the overall result, and highlights the metrics that deserve attention. The manual explorer lets you compare one exact workload by matching the same topology, middleware, communication mode, payload size, and process layout on both sides. A coverage view checks whether two runs contain the same tests before comparing them.
The dashboard displays latency, throughput, CPU usage, memory usage, and message reliability where available. It also preserves benchmark metadata including the ROS distribution, middleware, executor, benchmark commit, client library source, hardware platform, payload size, and process layout.
Default comparison view
Figure X. Default dashboard comparing the median Jazzy and median Lyrical benchmark runs.
Manual explorer
Figure Y. Manual explorer showing matching benchmark configurations for detailed investigation.
The public dashboard currently contains five Jazzy runs, five Lyrical runs, and one median summary for each distribution. Every benchmark in the supported matrix runs for 60 seconds. Containers are pinned to the same physical performance cores, and the Jazzy and Lyrical run order is alternated to reduce scheduling and thermal bias. The median summaries provide the primary comparison, while the individual runs remain available to inspect run-to-run variation. Results are published only after the full dataset has been validated.
The complete workflow from running benchmarks to exploring comparisons is working locally, and the dashboard is publicly available at performance.ammaar.lol. Current work focuses on improving repeated run summaries and handling incomplete upstream benchmark data.
I would like to thank Kimberly McGuire and Skyler Medeiros for their guidance, careful reviews, and feedback throughout the project. I am also grateful to OSRF, the ROS community, and everyone who answered questions and made me feel welcome in the community.
Work completed so far
- Built a single CLI workflow that prepares the benchmark project, runs it in Docker, and saves the results.
- Added a consistent result format so measurements from many benchmark files can be searched and compared together.
- Covered publish and subscribe tests as well as client and service tests, from 10-byte messages through
4 MiBmessages. - Added support for
Fast DDS,Cyclone DDS, and the availableZenohconfigurations. - Recorded the context needed for fair comparisons, including ROS version, middleware, executor, process layout, communication mode, source revisions, and machine platform.
- Added checks that reject missing or incompatible result files instead of silently presenting partial data as complete.
- Built guided, detailed, and manual Grafana views for comparing latency, throughput, CPU, memory, reliability, and test coverage.
- Replaced the confusing headline percentages with a plain language comparison result, a reason, and a suggested next action.
- Added safe container reuse and image reuse so repeated benchmarks do not rebuild large Docker images unnecessarily.
- Added optional CPU pinning to reduce interference on machines with different types of CPU cores.
- Added tests and documentation throughout the benchmark, parsing, exporting, and dashboard workflow.
- Deployed a public read only dashboard and documented how its code, data, archives, and services are maintained.
Work remaining
There is still time left in the GSoC period. The main remaining work is:
- Turn repeated run aggregation into a supported command that validates matching runs, excludes warm ups, records its sources, produces median results, and communicates variability clearly.
- Handle incomplete upstream output, including non finite values, missing latency files, unreliable message counters, multi-process synchronization, and pinned submodule revisions.
- Make result activation and archiving safer, and move duplicated knowledge about benchmark directory layouts into shared metadata.
- Fix Kilted support, improve setup documentation, and test the complete workflow on clean machines.
Support for rclpy, hosted automation, CI benchmarking, hosting under performance.ros2.org and a small local graphical launcher are possible stretch or post GSoC efforts. The CLI workflow would remain the main implementation so the project stays scriptable and reproducible.
I would value your feedback
If you work with ROS 2 performance, maintain a client library or middleware implementation, or are simply curious about how two ROS versions compare, please try the public dashboard. I would especially like to know whether the overall result makes sense without prior benchmark knowledge, whether you can find the scenario behind a warning, and what information would help you trust or question a comparison.
Feedback from both experienced ROS developers and people seeing these measurements for the first time would be very useful and appreciated from the depths of my heart. Thanks for reading.