ros_tap: zero config telemetry tap for any ROS robot

[pablothepenguin]

I’ve been working on a ROS 2 project where I’m moving fast on different control scripts. Every few hours I’m redeploying a new version to the robot, testing it, tweaking parameters, redeploying again. The loop is tight but the data problem kept getting worse.

Every time I deployed a new script I’d lose track of which run produced which data. I had terminal windows open with ros2 topic echo piping to files, but I’d forget to restart them after a redeploy, or I’d overwrite the previous run’s data, or I’d come back the next day and have no idea which bag file matched which version of the code. I tried writing wrapper scripts to timestamp everything but then I had a different wrapper for every robot and every experiment.

What I actually wanted was something that just sat on the network and recorded everything the robot published, regardless of what code

I was running on it. Something I didn’t have to redeploy or reconfigure every time I changed the robot’s software. Just a tap on the network that stayed running and captured every run cleanly.

So I built ros_tap. It’s a pip installable CLI that joins the DDS network (or queries a ROS 1 Master) from any machine, discovers all the nodes and topics automatically, and streams everything as JSONL. You don’t install anything on the robot. You don’t need ROS on the monitoring machine either, it uses CycloneDDS directly.

  pip install ros_tap

  ros_tap scan                          # see every robot and topic on the network

  ros_tap record                        # stream to stdout as JSONL

  ros_tap record -o ./data              # write to local files, auto rotated

  ros_tap record -o s3://bucket/prefix  # upload straight to S3

  ros_tap record -c power,imu,actuators # filter by category

It auto classifies topics into categories like power, actuators, imu, lidar, camera so you can filter without memorizing topic names.

It’s read only, never publishes anything, so it won’t interfere with whatever is already running on the robot. It pipes everything to our internal S3 buckets so the rest of the team can build analytics around this super quickly.

The thing that made the biggest difference for me was being able to just leave it running. Redeploy the robot, ros_tap picks up the new topics automatically. Every run gets its own timestamped data. No more lost experiments.

It supports both ROS 1 and ROS 2, and auto detects which is available on the network.

• PyPI: Client Challenge

• GitHub: GitHub - publu/ros_tap: Zero-config telemetry tap for any ROS 1 or ROS 2 robot on your network. No ROS install required. · GitHub

Would love feedback, especially on what topic patterns or categories people want auto detected. Right now it covers the common ones (battery, joints, IMU, lidar, camera, temperature) but I’m sure I’m missing some that are obvious to people working on different platforms.

[chfritz]

Interesting idea.

@pablothepenguin

discovers all the nodes and topics automatically, and streams everything as JSONL

This comes up every once in a while. Most people, me included, think that it’s not such a good idea to listen to all topics. Even if just locally on lo, it puts pressure on resources.

@pablothepenguin

It pipes everything to our internal S3 buckets

This can go wrong easily. What if you have an IMU that publishes at 1000Hz plus a 3d lidar that publishes a point cloud at 15 Hz? I hope you are ready for the Internet and S3 bill for that.

Instead of S3, or in addition to, you might also want to consider uploading into a robot-specific database like ReductStore (which recently went open-source). FYI, @AnthonyCvn.

@pablothepenguin

or I’d come back the next day and have no idea which bag file matched which version of the code

I don’t understand how your new tools solves this problem. The topics it listens to do not carry versioning information either.

[pablothepenguin]

@chfritz

Most people, me included, think that it’s not such a good idea to listen to all topics. Even if just locally on lo, it puts pressure on resources.

Good point. It doesn’t subscribe to everything by default in record mode. You can filter by category or topic name, so you’d typically run something like ros_tap record -c power,actuators,imu rather than capturing raw point clouds and camera feeds. The scan command discovers everything but that’s a one shot read of the topic graph, not a continuous subscription.

@chfritz

What if you have an IMU that publishes at 1000Hz plus a 3d lidar that publishes a point cloud at 15 Hz? I hope you are ready for the Internet and S3 bill for that.

S3 is one of three output options. Default is stdout, and local file output auto rotates into timestamped JSONL files. For high frequency sensors you’d definitely want local first and package it upstream in ~50mb files (what i calculated as optimal Cloudflare R2 bucket write/read pricing)

@chfritz

ReductStore

Hadn’t seen this! Will take a look

@chfritz

I don’t understand how your new tools solves this problem. The topics it listens to do not carry versioning information either.

You’re right that topics don’t carry that info. What helped me was that each recording session produces its own timestamped file, so when I redeploy and it picks up the new topic graph, there’s a clear boundary in the data. It’s not version tracking but it’s better than one big bag file that spans multiple deploys with no markers. Run metadata or deploy ID per session could be a better automated fit.

Honestly I’m using this in production myself and my goal is to make the lower level tooling solid enough that I don’t have to think about it while I’m focused on the actual robot work. If it just sits there and captures clean data every run without me touching it, that’s the win.