Ros2_pulse: per-topic rates and node liveness, including intra-process, on stock binaries

Projects
, , , ,
1

I run a ROS 2 planner stack on a Jetson AGX Orin and kept hitting the same wall: I want to know whether every topic is flowing at the rate it should and which nodes are alive, and the stock tools could not tell me without changing the system under test.

So I measured what watching a topic actually costs, and then built a probe that avoids it. Both are open source (Apache-2.0) and I would like your feedback.

pulse-top-demo

pulse-top --demo: the probe’s log, live. Sparklines per topic, intra-process rates, structured warnings with ages. There is also a 40-second video of the whole story.

What watching a topic costs today

We benchmarked the stock CLI against a 50 Hz publisher of 100 KB messages, with an in-process probe as the ruler (method and raw data):

  • ros2 topic hz on one topic: 7 % of a core.
  • ros2 topic echo on the same topic: 31 % of a core.
  • Pointing hz at an intra-process topic is worse than expensive: the publisher starts serializing every message for the new external subscriber, which raised the watched process’s CPU by 52 % in our runs. The act of looking changes the thing you are looking at.

And on Humble-class binaries, intra-process traffic (composable nodes carrying point clouds, for example) is invisible to hz, echo and the built-in topic statistics entirely (rclcpp#2911; fixed on rolling by rclcpp#3130, no Humble backport).

What ros2_pulse does instead

libros2_pulse.so is an LD_PRELOAD shim over the tracetools instrumentation layer that rclcpp already calls on every publish and every callback. It counts in-process with a lock-free hot path and appends per-window rates to a small rolling file. It needs no rebuild of your nodes, no root and no tracing session, and it adds no DDS traffic.

export LD_PRELOAD=libros2_pulse.so
ros2 launch your_stack your.launch.py
tail -f /tmp/topic_freq.<pid>.log

# ts_ns=1782887153899445923 window_s=5.000
TOPIC /scan 20.000000
RECV  /points inter=0.000000 intra=30.000000   # intra-process, invisible to other tools on Humble
NODE  /perception
WARN  TOPIC /scan hz=1.200000 expected=[18,22]

You can declare expected rates per topic and get WARN lines when a window violates them, opt into per-endpoint jitter (largest inter-arrival gap), and switch the output to JSON Lines for a Prometheus/OTel sidecar or a log shipper.

The cost, measured: the counting hot path is about 0.3 ns per message on a fixed endpoint (about two orders of magnitude under a single LTTng-UST tracepoint), and the whole probe is about 2 % of workload CPU on a deliberately harsh 4,900 msg/s stress, less on real graphs. On the Orin itself, the optional jitter clock read adds 51 ns per message. Full methodology, paired trials with error bars, in the benchmarks.

pulse-top

A terminal dashboard that tails the probe’s logs across processes: sparklines per topic, intra-process rates, warnings with ages, stall detection.

pip3 install ros2-pulse-top
pulse-top --demo   # try it without a robot

Install

The probe builds as a normal colcon package on Humble, Jazzy and Kilted (CI runs all three on stock ros:<distro> images):

cd ~/ros2_ws/src && git clone https://github.com/TanayK07/ros2_pulse.git
cd ~/ros2_ws && colcon build --packages-select ros2_pulse && source install/setup.bash

apt packages are on the way: rosdistro PRs for all three distros are open (humble, jazzy, kilted), so apt install ros-<distro>-ros2-pulse should work after the next sync.

Related work, and where this sits: CARET (Tier IV) hooks the same tracetools layer, which was a useful independent validation of the mechanism, but targets deep offline latency analysis with LTTng and a forked rclcpp. ros2_tracing is also offline-first. This probe is the always-on, read-it-now complement, not a replacement for either.

It has been running against a 77-node production planner stack on an Orin. I would love to hear what breaks on your graphs, and whether the expected-rate spec covers what you would actually alert on.

8 Likes
2

Wow! This, together with ros2_probe looks like the fantastic ROS 2 observability tool we’ve been missing so far!

Would it be possible to publish the measured data on the /statistics topic instesad of just writing them down in a file?

1 Like
3

The last observability wish I have is “this pub-sub link is dropping messages on the way”. Anyone? :slight_smile:

1 Like
4

Yes if you go through the docs, it gives you that option such that you can write it on your respective files for your telemetry stack and dashboards to pick it up

5

Yeah, you don’t actually need ros2_probe after you use this :eyes:

6

Hi Guys would really appreciate a star on this repo :star:

7

Very cool!
Been using greenwave_monitor until now, GitHub - NVIDIA-ISAAC-ROS/greenwave_monitor: ROS 2 Topic Monitoring · GitHub

Pulse seems to do more, going to try it out!

1 Like
8

Congrats on the release, @TanayK07. The intra-process side is a genuinely useful
thing to have on stock Humble binaries, and rclcpp#2911 has been a real gap for
anyone running composable nodes. Nice to see someone go after it.

Since @peci1 mentioned the two projects together, it might help to spell out
where each of us sits, because I think we are working on different halves of the
same problem.

ros2_pulse observes inside the process. It interposes the tracetools symbols
that rclcpp already calls, so it counts publishes and callbacks at the
application boundary. That is exactly what lets it see intra-process delivery,
which never touches a socket.

ros2probe observes below the middleware. An eBPF socket filter reads a kernel
copy of the RTPS/Zenoh packets, so nothing is loaded into the target process and
nothing joins the DDS graph. That is what lets it answer a different set of
questions:

  • end-to-end delay and per-topic bandwidth, not just rate
  • the loss the real subscriber saw, reconstructed from RTPS sequence numbers
  • decoded message content (rp topic echo) and MCAP recording
    (rp bag record) as a rosbag2 replacement
  • anything on the wire, including rclpy nodes and non-ROS DDS participants
  • attaching to, and detaching from, a stack that is already running, without
    restarting it

So the two answer different questions. “Is /points flowing at 30 Hz inside this
composable container” is something pulse answers and we structurally cannot.
“How long did that message take from publish to callback, how much bandwidth is
it using, what was in it, and did the subscriber actually receive it” is what we
built for.

@peci1, on your second wish, “this pub-sub link is dropping messages on the way”:
that one is squarely in our lane and it is next on our list. Because we read the
RTPS sequence numbers off the wire, we can report the gap the real subscriber
experienced rather than the gap an extra observer subscriber would have
experienced. Per-link sequence-gap reporting is what we are building toward.

One more thing worth saying out loud, since ros2_pulse lands right next to it.
Today ros2probe falls back to a short-lived shadow subscriber for shared-memory
topics, and that fallback gives up our probe-effect-free guarantee for exactly
the reason @TanayK07 measured, namely that the publisher starts serializing for
the new reader. We are not happy with that and we are replacing it. Bringing
shared memory under the same guarantee the wire already has, no participant
added and nothing injected into the target process, is our current priority.
Intra-process delivery is on the same track. I will report back on this thread
when it lands.

That sits alongside the layer-by-layer latency measurement tool I mentioned in
our thread a few days ago. The goal is a per-layer breakdown of publish-to-
callback latency under the same constraints ros2probe already holds. If that
sounds useful to your stacks, I would like to hear what you would want to see
separated out first.

Also, for anyone following this thread, ros2probe v0.2.0 added rmw_zenoh support
over both TCP and UDP, alongside Fast DDS and Cyclone DDS.

Code: GitHub - csi-dgist/ros2probe: Host-level observability for ROS 2 middleware traffic, without creating any ROS 2 subscriptions. · GitHub
Paper: [2606.10746] ros2probe: Non-intrusive, Kernel-selective Observability for Robot Operating System 2 Middleware