We analyzed the glass-to-glass latency of streaming video from robots to the web using WebRTC. Typical, total latency for remote streaming is 150-180 ms but how does this break down?
Tl;dr:
The vast majority of latency actually comes from the camera itself and the USB bus (~100 ms).
H264 encoding and decoding add around 10 ms each (or less).
WebRTC only adds around 10 ms of latency for remote streaming (jitter buffers).
The rest is due to static network delay (“ping timing”, speed of light).
Very nice analysis, I recently stumbled upon something similar, when I wanted to do “high” frequency object following, I got on average around 90ms at the end of the pipeline, I believe the ros2 delay command measures something similar? Mine was ~40ms with realsense
ros2 topic delay /camera_info_rect
WARNING: topic [/camera_info_rect] does not appear to be published yet
average delay: 0.084
min: 0.033s max: 0.352s std dev: 0.09249s window: 27
average delay: 0.058
min: 0.033s max: 0.352s std dev: 0.06823s window: 57
average delay: 0.050
min: 0.033s max: 0.352s std dev: 0.05630s window: 87
average delay: 0.046
min: 0.033s max: 0.352s std dev: 0.04903s window: 117
average delay: 0.043
min: 0.033s max: 0.352s std dev: 0.04400s window: 147
average delay: 0.042
min: 0.032s max: 0.352s std dev: 0.04025s window: 177
average delay: 0.041
min: 0.032s max: 0.352s std dev: 0.03734s window: 207
average delay: 0.040
min: 0.032s max: 0.352s std dev: 0.03497s window: 237
average delay: 0.039
min: 0.032s max: 0.352s std dev: 0.03300s window: 267
average delay: 0.039
min: 0.032s max: 0.352s std dev: 0.03133s window: 297
I only used USB cameras with ROS so far, do other interfaces like CSI or ethernet have lower latency? I know Nvidia is doing Holoscan which should use some FPGA + GPU acceleration to get lower latency - it would be nice if someone shared their latency camera optimization experiences
I think the main conclusion from this analysis is don’t use USB cameras MIPI/CSI/GMSL have better latencies. GigE cameras have latencies adequate to the gigabit Ethernet link. The latency of Ethernet processing itself is pretty minimal, but you should always check for bufferbloat.
If I’m making 300$ home/edu robot, adding 500$ GMSL camera will kill my project, on the other hand adding few thousands to an autonomous bus/tram budget is a drop in the bucket…
Also depth cameras like Realsense are doing nontrivial processing on device, while GMSL seems to send raw data and then you need to do processing yourself, on the other hand, you might do some inference on raw values to skip that entirely.
Seems like GMSL is really cool latency wise, it would be nice to have some numbers
I have mainly USB cams like Realsense/Oak, but I can test RPi cam on Jetson/Pi 5 with CSI.
Ethernet cameras are also interesting too, but squishing switch/cables on a small robot could be big challenge too.
I don’t have any fancy numbers, but I get ros2 topic delay ~50 ms for FullHD@30 stream from a GMSL camera (don’t compare it with Christian’s screenshot - that was for a VGA stream).
I don’t think in-band time-measurements like ros2 topic delay can account for delays that happen between the camera lens and the first compute unit, where a header timestamp could be written, so I think it measures something different.
I actually have one on my desk right now for a customer. The one I have, Oak D S2 still connects over USB and I still see the same 100 ms “base” latency when I measure using the infinite mirror setup described in the article, like this (but with the Oak):
I saw that in your article you mention NVDEC/NVENC, do you have any experience with these? I was also thinking about writing some thin WebRTC teleop for my robots, and I have feeling that doing sw decoding/encoding would consume lot of resources on devices like RPi. Jetson on the other hand have dedicated HW for these. So maybe on desktop workstation enc/dec is very cheap, but on low power devices using some HW enc/dec is a must?
Btw I had very similar discussion, I recently upgraded my WiFi M.2 card on Jetson to WiFi 7 to improve throughput/jitter/latency, and someone said that they will just use WiFi 7 USB stick I guess they will certainly see some increased latency just from using USB if they stream large amounts of data.
Yes, definitely. Our webrtc-video and remote-teleop packages support the most common hardware acceleration platforms (Nvidia nvenc, Intel va-api, and RockChip mpp), both on SoCs like Jetsons and Orange Pis, but also NUCs and desktop GPUs. And yes, it’s a must for sure, especially when streaming multiple cameras.
Feel free to try out our modules (free to try). There are a lot of things to consider even besides hardware acceleration, e.g., congestion control, packet loss mitigation, handling multiple streams, etc.
At least from what I was told by the person who integrated the cameras, GMSL should sync cameras with system time and they should thus timestamp capture with system time (i.e. not timestamping on arrival to ROS driver, but really stamping the moment of capture). Same with properly configured GigE cameras synced over PTP. So this delay should really be the glass-to-ros stamp (microsecond accuracy hopefully).
RPi also has a HW encoder for H.264, nowadays probably via OMX or some similar framework (should be integrated in the RPi-distributed GStreamer). The encoder on RPi4 can encode FullHD at almost 30 FPS into H.264.
Also, regarding basically all HW encoders, you have to be prepared that the output and configurations of them can be quite limited. E.g. the Ryzen integrated GPU in my laptop cannot encode B-frames in H.264/5. They are also often optimized for speed and not quality.
We are using a Radxa Zero 3E (though they were basically sold out last time we tried to get more), and they have mpp hardware encoding.
I can recommend them performance-wise, but they run quite hot (with active cooling, it’s good, though), SD card performance isn’t great, and they are easily killed by overvoltages.
Still the best small SBC we have found so far.
Using
reading out the MIPI camera at 720p and sending it to our operator station using WebRTC (with a local signaling server hosted by the camera server), we get a capture to display a latency of around 30ms.
The MIPI latency is unknown, but it’s more than fast enough for teleoperation.
A small note: this camera server is not competing with Foxglove or Transitive, but is intended for local setups that don’t want or need an internet connection.
It makes setting up camera streams for ROS with low-latency teleop streams much easier by automatically building and supervising GStreamer pipelines, with hardware acceleration when available.
@chfritz I do encounter one issue with gstreamer’s webrtcbin, though, sometimes the latency rises to unusable levels of multiple seconds. Have you ever encountered something like that?
SRT doesn’t have that issue, and I’ve determined that it’s definitely after the encoding.
That’s impressive. Thanks for sharing. How did you measure this?
Sounds like you haven’t implemented a congestion control mechanism yet, i.e., a dynamic bitrate controller that constantly estimates available network bandwidth and adjust the encoding bitrate to keep latency low. May not be required as much on local networks, but it’s an absolute must for remote streaming. The other possible culprit could be packet loss. If a keyframe packet is lost and you didn’t implement any mitigation strategy, then the receiver will need to wait for the next keyframe to arrive, which, depending on your configuration can easily be a few seconds or more.
The server attaches a timestamp in a probe when v4l2 produces a buffer. This timestamp is preserved during the entire processing as buffer meta (using passthrough probes when going through encoders) and transmitted as a custom RTP extension.
The receiving computer is synchronized using chrony, extracts the timestamp and takes the difference of the timepoint when the buffer has been decoded and is handed over to QML as video frame for displaying to the attached timestamp.
I’ve investigated again due to your question and noticed that I could improve this a bit more by extracting the monotonic v4l2 timestamp and converting that to a ROS 2 now system clock timestamp.
Should not make much of a difference, though, and I believe it would still not include the transfer time from sensor to the kernel.
Yes, for simplicity, I have not added dynamic bitrates at the moment.
It’s not just on the receiver side, though. It looks like it’s stuck in the pipeline on the sender’s side.
The processing time, measured using the same timestamp, keeps increasing, as if webrtcbin were blocking.
There are queues with time limits before the webrtcbin, which should prevent stale data from accumulating, so I believe it must happen inside the webrtcbin.
In that case, I think, we are comparing two different things. As the image in the original article indicates, up to 80 ms are lost between photons hitting the camera lens and the USB subsystem on the computer receiving the corresponding video frame. I’m absolutely willing to believe MIPI is faster, but I’d like to rule out s and s.
Any chance you could measure again using the infinite mirror setup? For simplicity you can run while true; do echo -ne "$(date +%H:%M:%S:%N)\r"; done in a terminal to display a high-precision clock on the screen at which you point the camera and display the camera image (as suggested by this, really great article by Ridge Run).
but you for sure need some 
I guess they will certainly see some increased latency just from using USB if they stream large amounts of data.
s and 
s.