Ros2_pulse: per-topic rates and node liveness, including intra-process, on stock binaries

Projects
, , , ,
1

I run a ROS 2 planner stack on a Jetson AGX Orin and kept hitting the same wall: I want to know whether every topic is flowing at the rate it should and which nodes are alive, and the stock tools could not tell me without changing the system under test.

So I measured what watching a topic actually costs, and then built a probe that avoids it. Both are open source (Apache-2.0) and I would like your feedback.

pulse-top-demo

pulse-top --demo: the probe’s log, live. Sparklines per topic, intra-process rates, structured warnings with ages. There is also a 40-second video of the whole story.

What watching a topic costs today

We benchmarked the stock CLI against a 50 Hz publisher of 100 KB messages, with an in-process probe as the ruler (method and raw data):

  • ros2 topic hz on one topic: 7 % of a core.
  • ros2 topic echo on the same topic: 31 % of a core.
  • Pointing hz at an intra-process topic is worse than expensive: the publisher starts serializing every message for the new external subscriber, which raised the watched process’s CPU by 52 % in our runs. The act of looking changes the thing you are looking at.

And on Humble-class binaries, intra-process traffic (composable nodes carrying point clouds, for example) is invisible to hz, echo and the built-in topic statistics entirely (rclcpp#2911; fixed on rolling by rclcpp#3130, no Humble backport).

What ros2_pulse does instead

libros2_pulse.so is an LD_PRELOAD shim over the tracetools instrumentation layer that rclcpp already calls on every publish and every callback. It counts in-process with a lock-free hot path and appends per-window rates to a small rolling file. It needs no rebuild of your nodes, no root and no tracing session, and it adds no DDS traffic.

export LD_PRELOAD=libros2_pulse.so
ros2 launch your_stack your.launch.py
tail -f /tmp/topic_freq.<pid>.log

# ts_ns=1782887153899445923 window_s=5.000
TOPIC /scan 20.000000
RECV  /points inter=0.000000 intra=30.000000   # intra-process, invisible to other tools on Humble
NODE  /perception
WARN  TOPIC /scan hz=1.200000 expected=[18,22]

You can declare expected rates per topic and get WARN lines when a window violates them, opt into per-endpoint jitter (largest inter-arrival gap), and switch the output to JSON Lines for a Prometheus/OTel sidecar or a log shipper.

The cost, measured: the counting hot path is about 0.3 ns per message on a fixed endpoint (about two orders of magnitude under a single LTTng-UST tracepoint), and the whole probe is about 2 % of workload CPU on a deliberately harsh 4,900 msg/s stress, less on real graphs. On the Orin itself, the optional jitter clock read adds 51 ns per message. Full methodology, paired trials with error bars, in the benchmarks.

pulse-top

A terminal dashboard that tails the probe’s logs across processes: sparklines per topic, intra-process rates, warnings with ages, stall detection.

pip3 install ros2-pulse-top
pulse-top --demo   # try it without a robot

Install

The probe builds as a normal colcon package on Humble, Jazzy and Kilted (CI runs all three on stock ros:<distro> images):

cd ~/ros2_ws/src && git clone https://github.com/TanayK07/ros2_pulse.git
cd ~/ros2_ws && colcon build --packages-select ros2_pulse && source install/setup.bash

apt packages are on the way: rosdistro PRs for all three distros are open (humble, jazzy, kilted), so apt install ros-<distro>-ros2-pulse should work after the next sync.

Related work, and where this sits: CARET (Tier IV) hooks the same tracetools layer, which was a useful independent validation of the mechanism, but targets deep offline latency analysis with LTTng and a forked rclcpp. ros2_tracing is also offline-first. This probe is the always-on, read-it-now complement, not a replacement for either.

It has been running against a 77-node production planner stack on an Orin. I would love to hear what breaks on your graphs, and whether the expected-rate spec covers what you would actually alert on.

13 Likes
Ros2_pulse: topic rates on a Grafana dashboard, now counting rclpy nodes and loaned messages
ROS News for the Week of August 24th, 2026
ROS News for the Week of September 7th, 2026
Simple C++ node to get more accurate topic frequency
2

Wow! This, together with ros2_probe looks like the fantastic ROS 2 observability tool we’ve been missing so far!

Would it be possible to publish the measured data on the /statistics topic instesad of just writing them down in a file?

1 Like
3

The last observability wish I have is “this pub-sub link is dropping messages on the way”. Anyone? :slight_smile:

1 Like
4

Yes if you go through the docs, it gives you that option such that you can write it on your respective files for your telemetry stack and dashboards to pick it up

5

Yeah, you don’t actually need ros2_probe after you use this :eyes:

6

Hi Guys would really appreciate a star on this repo :star:

7

Very cool!
Been using greenwave_monitor until now, GitHub - NVIDIA-ISAAC-ROS/greenwave_monitor: ROS 2 Topic Monitoring · GitHub

Pulse seems to do more, going to try it out!

1 Like
8

Congrats on the release, @TanayK07. The intra-process side is a genuinely useful
thing to have on stock Humble binaries, and rclcpp#2911 has been a real gap for
anyone running composable nodes. Nice to see someone go after it.

Since @peci1 mentioned the two projects together, it might help to spell out
where each of us sits, because I think we are working on different halves of the
same problem.

ros2_pulse observes inside the process. It interposes the tracetools symbols
that rclcpp already calls, so it counts publishes and callbacks at the
application boundary. That is exactly what lets it see intra-process delivery,
which never touches a socket.

ros2probe observes below the middleware. An eBPF socket filter reads a kernel
copy of the RTPS/Zenoh packets, so nothing is loaded into the target process and
nothing joins the DDS graph. That is what lets it answer a different set of
questions:

  • end-to-end delay and per-topic bandwidth, not just rate
  • the loss the real subscriber saw, reconstructed from RTPS sequence numbers
  • decoded message content (rp topic echo) and MCAP recording
    (rp bag record) as a rosbag2 replacement
  • anything on the wire, including rclpy nodes and non-ROS DDS participants
  • attaching to, and detaching from, a stack that is already running, without
    restarting it

So the two answer different questions. “Is /points flowing at 30 Hz inside this
composable container” is something pulse answers and we structurally cannot.
“How long did that message take from publish to callback, how much bandwidth is
it using, what was in it, and did the subscriber actually receive it” is what we
built for.

@peci1, on your second wish, “this pub-sub link is dropping messages on the way”:
that one is squarely in our lane and it is next on our list. Because we read the
RTPS sequence numbers off the wire, we can report the gap the real subscriber
experienced rather than the gap an extra observer subscriber would have
experienced. Per-link sequence-gap reporting is what we are building toward.

One more thing worth saying out loud, since ros2_pulse lands right next to it.
Today ros2probe falls back to a short-lived shadow subscriber for shared-memory
topics, and that fallback gives up our probe-effect-free guarantee for exactly
the reason @TanayK07 measured, namely that the publisher starts serializing for
the new reader. We are not happy with that and we are replacing it. Bringing
shared memory under the same guarantee the wire already has, no participant
added and nothing injected into the target process, is our current priority.
Intra-process delivery is on the same track. I will report back on this thread
when it lands.

That sits alongside the layer-by-layer latency measurement tool I mentioned in
our thread a few days ago. The goal is a per-layer breakdown of publish-to-
callback latency under the same constraints ros2probe already holds. If that
sounds useful to your stacks, I would like to hear what you would want to see
separated out first.

Also, for anyone following this thread, ros2probe v0.2.0 added rmw_zenoh support
over both TCP and UDP, alongside Fast DDS and Cyclone DDS.

Code: GitHub - csi-dgist/ros2probe: Host-level observability for ROS 2 middleware traffic, without creating any ROS 2 subscriptions. · GitHub
Paper: [2606.10746] ros2probe: Non-intrusive, Kernel-selective Observability for Robot Operating System 2 Middleware

3 Likes
9

@sanghoon_lee Thanks for spelling it out, and I was too quick in my reply above. Your split is right: pulse counts at the application boundary inside the process, ros2probe reads the wire below the middleware, and neither can do the other’s job. They also meet in one useful place: you can see that a packet reached the subscriber’s reader, pulse can see whether the callback ran. Same message, two ends. On which latency layer to separate first: for my stack it would be the time between the message landing in the reader and the callback starting, because that executor wait is what shows up in pulse as a rate dip that is not a drop. I’ll follow the thread for the shared memory work.

1 Like
10

Happy to see you’re re-using the tracetools instrumentation! This is exactly why we decoupled the instrumentation from the actual tracepoints.

And I totally agree that you shouldn’t measure/collect performance-related data from a framework using the framework itself (i.e., pub/sub) if you really want valid data, although it’s good enough in a lot of cases.

I would’ve loved to see something like this completely using LTTng*, though! I feel like, if you try to make this system-wide and add more advanced features (e.g., control how much & when you write the data to disk, support multiple/distributed systems), you’ll basically end up reimplementing a tracer. But I can definitely appreciate small & simple tools with a single goal like other Unix-style tools.

*you can definitely do online trace data processing (ros2/ros2_tracing#149)

2 Likes
11

@peci1 Correcting my earlier reply: it could not publish on /statistics, only write a file. I was wrong about that.

It can now. v0.5.0 adds pulse_bridge, a sidecar that tails the probe’s jsonl log and republishes every window on /statistics as statistics_msgs/MetricsMessage, in the same shape the built-in topic statistics use (unit ms, AVERAGE = message period, SAMPLE_COUNT, MAXIMUM = largest gap when jitter tracking is on). Anything that already reads that topic gets the probe’s numbers, intra-process included.

export ROS_TOPIC_STATS_FORMAT=jsonl    # on the probed processes
ros2 run ros2_pulse pulse_bridge        # one per machine, follows $TMPDIR/topic_freq.*.log
ros2 topic echo /statistics

It is a separate process on purpose: the probe still never joins the DDS graph, so the thing being measured is not the thing publishing the measurement. Release: Release v0.5.0 · TanayK07/ros2_pulse · GitHub (the rosdistro entries merged on 09-14, so apt follows the next sync). Details: GitHub - TanayK07/ros2_pulse: Near-zero-overhead ROS 2 topic/node frequency probe (inter + intra process) via an LD_PRELOAD tracetools shim. · GitHub

On “this pub-sub link is dropping messages”: pulse cannot answer that. It counts at the application boundary, so it sees a callback run or not run, but not why. Loss on the wire is what ros2probe reconstructs from RTPS sequence numbers, and that is the right layer for it.

1 Like
12

@christophebedard Thank you, and the decoupling is what made this possible: the interposers are only viable because the ros_trace_* calls are in the binaries whether or not a tracer is attached.

Why not LTTng for this: on Humble the stock tracetools has no lttng-ust backend, so an LTTng path needs ROS rebuilt, and “stock binaries, no rebuild” was the constraint I was solving for. On Jazzy and later, where it works out of the box, ros2_tracing with online processing (#149) is a legitimate way to get the same rates, and I would point people there when they already run a session.

Your prediction about ending up reimplementing a tracer is fair, and it is the line I am trying not to cross. The in-process part is a counter and stays one. The one thing added since this thread, a /statistics republisher, is deliberately a separate process that reads the file, so the probe itself does not grow.

One question, since the whole approach rests on it: do you consider the set of instrumentation points stable across distros? The Humble/Jazzy/Kilted matrix works today, and I would like to know whether that is by design or by luck.

1 Like
13

Two updates since the last post.

pulse-top 0.4.0 adds pulse-export. It serves the probe’s windows as Prometheus metrics on :9464 and can also push OTLP/HTTP, so you can put a robot on a Grafana dashboard without touching its build. pip install ros2-pulse-top, and there’s a docker compose with a dashboard in examples/grafana: ros2_pulse/examples/grafana at main · TanayK07/ros2_pulse · GitHub

Also fixed a bug worth knowing about if you ever freeze a process to test the gap warns. After a SIGSTOP/SIGCONT, the flush timer replayed every missed tick back to back, so the latest window showed a near-zero gap instead of the stall. It now re-phases after a stall, and a 5 s freeze shows up as a 5 s gap (#62, on main, in the next release).

1 Like
14

The official policy is the same as the rest of the ROS 2 core: we only guarantee stability within the same distro and use a tick-tock deprecation strategy, usually across consecutive distros (although deprecation is hard to do at this level!). However, we did still put effort into keeping the tracepoints stable across distros just to make trace data processing easier.

Most of the changes have added tracepoints instead of removing them. I did “deprecate” tracepoint arguments/fields because I found that some of them were redundant, but I did not actually remove them. We’ll probably remove those eventually.

Note that there are mechanisms to handle differences across distros, just in case you’re not aware:

  1. In source code: include tracetools/version.h (generated using ament_cmake_gen_version_h) and #ifdef based on the tracetools version.
  2. On the trace processing side: you can get the same version string from the rcl_init tracepoint’s version field. This way you don’t need the original system that produced the trace.
  3. At build-time: add a <build_depend>ros_environment</build_depend> to your package and use $ENV{ROS_DISTRO} in CMake, e.g., if($ENV{ROS_DISTRO} MATCHES "rolling").
1 Like
15

thanks, super helpful. one catch though, the probe can’t see that version field: tracetools adds it inside its own event and the hook only ever gets the context handle (same on humble, jazzy and rolling). so i went the other way, it now finds the libtracetools.so behind the hooks, reads the version from its package.xml, puts it in the startup banner and warns if it’s a major CI doesn’t run (4.x, 8.x, 9.x): feat(probe): report the tracetools version at startup and warn on an untested one by TanayK07 · Pull Request #66 · TanayK07/ros2_pulse · GitHub

good to know args get deprecated and not dropped, the probe mostly reads handles so that should hold up