How do you simulate latency?

Gazebo Gazebo General
1

Did someone already do something regarding simulating latency (or even jitter) in sensor data and control commands?

I think the obvious approach when you work with ROS is to implement something like topic_tools/delay and plug it behind the bridge. However, that seems to massively increase the number of topics, making it very error-prone to pick the right one…

I can imagine a similar mechanism on the gz-transport side, that would at least help with the number of ROS topics. But isn’t there something even closer to the simulation?

2 Likes
2

Are you talking about modeling latency for RL training? If so, we don’t use ROS for that, we have our latencies baked into the model inputs and outputs.

3

Not necessarily RL training. I’m talking about CI pipelines, PID tuning and general use.

So your model basically has a queue for each input/output?

4

Whichever mechanism you end up with, one thing I’d really push for is to seed the jitter and log the actual delay injected per message, not just the distribution you asked for. I build replay tooling for robot fleets and this one bit me hard.

Unseeded injected noise gives you failures you can see but can’t get back. The way I ended up doing it is that every injected fault gets recorded as part of the run itself, so when some cursed interleaving trips CI, that exact run replays instead of turning into a flaky ghost.

Doesn’t answer your topology question, sorry, but whichever way you delay things, making the injection part of the record pays off the first time something actually breaks.

2 Likes
5

For CI and PID tuning specifically, I have had better luck putting the delay below the middleware rather than inside the graph. A netem qdisc on the interface, or on a veth pair if you only want to shape one direction, gives you delay, jitter with a distribution, reordering and loss in one place, and it never touches your topic list. That is closer to the wire than a delay node, which for a lot of these questions is the same as closer to the simulation.

The reason I moved off delay nodes is not only the topic explosion. A delay node holds an already-delivered message and republishes it later, so what you get is a shifted timestamp. It does not reproduce what the transport does under those conditions. DDS reliability is a heartbeat and NACK conversation, and once you add real latency and loss, retransmit timing, history depth and missed deadlines start driving behavior. Those are usually the things that bite in the field, and a delay node hides all of them.

One caveat that surprises people the first time: netem shapes wall clock. If your control loop runs on /clock and the sim is not real-time, your shaped 100 ms is 100 ms of wall clock, not of sim time. For PID tuning that is usually what you want, but only if you run the sim real-time locked. Worth deciding up front which clock the tuning should be honest to.

Saketh’s point above is the one I would not skip. netem loss and jitter are pseudorandom, so a bare profile gives you a distribution, not a repeatable run, which is useless as a CI gate. Newer kernels let you pin the netem seed, and I keep two profiles: a fixed-delay, zero-jitter, pinned-seed one that gates the build, and a nastier randomized one that runs nightly and is allowed to fail loudly. Recording the injected fault into the run itself, the way he describes, is the more robust version of that, and the two compose: shape at the qdisc, record what you shaped.

For context, I work on remote operations for ROS 2 fleets across links I do not control, so degraded-link behavior is most of what I stare at.

3 Likes
Making a degraded-network matrix a required CI gate, and what did not reproduce
6

Thanks a lot for the great ideas! I assume these techniques do not play well with intraprocess comms, right? I usually like to run the most demanding nodes in the same container as Gazebo bridge and enable intraprocess for that…

7

Yeah, good instinct, and it’s worth being precise about why. netem only touches traffic that actually crosses a network interface, and rclcpp intra-process never does. Those messages just move by pointer, no serialization, no NIC, so there’s nothing for a qdisc to grab. Shared-memory DDS transports like iceoryx are the same deal. Plain DDS over loopback is the one exception, since it does ride lo.

For your setup that’s the right behavior, not a limitation. The nodes you’re co-locating in the bridge container really are on the same box at basically zero latency, so faking delay between them would just lie to your controller.

So I’d shape at the boundary instead. Take the one hop you actually care about, bridge to a remote consumer, or robot to operator, and push it across a process or netns boundary so it rides the real transport. Then netem that interface. Everything you keep intraprocess stays fast, and only the link you care about gets shaped.

One netns gotcha worth flagging: discovery has to cross that boundary too. Make sure multicast or your discovery config reaches across, or the endpoints just never find each other and it looks like a shaping bug when it isn’t.

8

Nope. Acquiring an image via a camera actually takes at least a few (tens of) ms, then transporting the image to the host, and only then the ROS message is formed. This is the latency I care about the most. Gazebo is like a camera shooting at 1/inf time with 0 transport and processing delay.

9

Ha, fair, I answered the wrong latency. You’re talking about the sensor pipeline, not the network hop. A real camera has exposure plus readout plus bus transport before a ROS message even exists, and Gazebo hands you the frame at t=0 with none of that. That is a real delay worth modeling, and it is a different animal from the transport shaping I was describing.

Here is the part that flips my earlier answer: for sensor acquisition latency a delay node is the right tool, not the wrong one. What you want is a pure time-shift of an already-formed message, which is exactly what topic_tools/delay does. My “delay nodes hide transport behavior” point only applies when you are modeling what the network does to a message in flight. That is not your problem. Yours is “this message should exist later than the sim thinks,” and a delay expresses that honestly.

The one detail that is the whole game: the header stamp. On real hardware the frame lands tens of ms late but stamped at capture time, so it is both delayed and already old when your estimator sees it. Check that whatever you use preserves the original stamp instead of re-stamping on republish, because the re-stamp erases the exact thing that makes late sensor data hard to handle.

And do the delay in sim time off /clock rather than wall clock, so a non-realtime run still gives you the right number.

2 Likes
10

Yeah, thanks for confirming my thoughts :slight_smile:

Some sensors like GigE cameras could actually be a combination of both latencies, but I wouldn’t expand on that too much. It’d seem more useful to me to just increase the jitter of the acquisition delays.

When I let my thoughts run really far away, I thought: why should there be a single /clock ? In reality, each computer/sensor has it own clock running at its own pace. So, in theory, you could wire every timestamping sensor in the sim to a specific clock :slight_smile: But hey, I don’t think I’ll pursue this road further :smiley:

2 Likes
11

We simulate latency at 3 points

  • Input cmd to the motors (we have a know transmission delay here)
  • Encoder feedback from the motors (same as above)
  • Images from our depth cameras (transmission delay)

We do this by using custom gazebo plugins to either delay the applying of the incoming cmds, or by delaying the publishing of the feedback and images.

As we adjust the timestamps in the real hardware drivers to compensate for the static transmission delay, and use ptp for the image sensors we publish with correct timestamps but delayed, as this models our real system as close as possible.

The effect of introducing delay is actually quite visible, as you can observer overshooting in the path following etc if you turn it on (and deactivate the compensation for it).

1 Like
12

Great, thanks for sharing! Are these Gazebo plugins somewhere public?

13

@peci1 If your goal is to simulate latency so that you can validate policy alignment and robustness(RL or otherwise), you can try this open-source project from QUT which injects different types of jitter, latency and dropped frames: anicut-ai/embodied-sync

14

If running your ROS nodes in a docker container for these tests is an option, then I highly recommend the linux built-in tc command to simulate various network conditions. We, for instance, use the following to test our webrtc video streaming solution:

tc qdisc replace dev eth0 root netem delay 400000 rate 2000kbit loss random 50

which simulates:

  • 400 ms delay
  • 2 Mbit/s bandwidth limit, and
  • 50 % packet loss.

See man tc-netem for details and search for jitter.

PS: should work on lo just the same.

1 Like