My first implementation is now complete, and I believe anyone using Linux with ROS 2 Lyrical or later should be able to try it. The backend plugin is memfdaccording to memfd_create used to create sharable memory with a file descriptor.
Benchmark results
I compared the latency of several backend and transport combinations using sensor_msgs/Image.
rosidl::Bufferbackend- default CPU
- this memfd backend
- transport setting
- inter-process
- intra-process
This gives four configurations:
- Inter CPU — normal DDS communication
- Inter SHM — the path introduced and optimized by this implementation
- Intra CPU — the ordinary optimized intra-process path
- Intra SHM — the ordinary optimized intra-process path plus the buffer backend overhead
Latency was measured:
- from immediately before
publish() - to immediately after reading the first byte of the buffer
Here is the results of that combinations with various Payload from 64 B to 16 MiB.
Here are the results for payload sizes ranging from 64 B to 16 MiB. The values are median / p99 latency in microseconds.
| Payload | Inter CPU | Inter SHM | Intra CPU | Intra SHM |
|---|---|---|---|---|
| 64 B | 636.3 / 726.9 | 951.4 / 1,091.2 | 144.5 / 179.9 | 114.8 / 173.8 |
| 1 KiB | 641.5 / 719.7 | 942.3 / 1,060.1 | 120.4 / 174.2 | 115.7 / 162.4 |
| 4 KiB | 667.8 / 753.0 | 927.2 / 1,045.0 | 118.4 / 166.0 | 121.5 / 169.3 |
| 16 KiB | 677.1 / 758.9 | 921.5 / 1,057.8 | 123.4 / 176.0 | 118.3 / 167.6 |
| 64 KiB | 687.6 / 776.9 | 958.8 / 1,107.0 | 124.6 / 174.8 | 123.1 / 168.2 |
| 256 KiB | 1,009.1 / 1,153.5 | 1,033.0 / 1,128.8 | 113.8 / 138.6 | 116.1 / 136.5 |
| 1 MiB | 11,700.0 / 13,774.4 | 1,172.9 / 1,298.2 | 118.4 / 128.7 | 119.8 / 132.9 |
| 4 MiB | 15,407.8 / 16,025.3 | 1,700.0 / 1,822.6 | 108.2 / 117.2 | 131.3 / 152.5 |
| 16 MiB | 14,879.5 / 16,019.0 | 2,289.5 / 2,718.5 | 37.1 / 78.9 | 112.7 / 127.5 |
When comparing the Inter CPU and Inter SHM results at small payload sizes, you may notice that Inter SHM has an additional overhead of about 300 microseconds. I suspect that this overhead mainly comes from creating the buffer descriptor from the buffer.
The Intra CPU and Intra SHM results support this interpretation: there is almost no difference between them, suggesting that using the SHM-backed buffer itself introduces very little overhead.
The latency of Inter SHM starts to increase above 256 KiB, even though the descriptor size remains almost constant. This was surprising, but I found that the increase came from the rmw_fastrtps_cpp implementation, and I submitted a patch to address it:
In my environment, the crossover point between Inter CPU and Inter SHM was around 256 KiB.
After applying the patch
Here are the results after applying the patch. In this case, the latency of Inter SHM remains almost constant.
This is expected behavior: with a memfd-backed buffer, the payload itself does not need to be serialized, and the descriptor size is almost constant regardless of the payload size.
| Payload | Inter CPU | Inter SHM | Intra CPU | Intra SHM |
|---|---|---|---|---|
| 64 B | 680.1 / 754.4 | 944.0 / 1,073.8 | 120.7 / 174.7 | 118.2 / 160.4 |
| 1 KiB | 633.8 / 741.7 | 942.1 / 1,059.1 | 121.1 / 171.6 | 116.1 / 151.6 |
| 4 KiB | 666.1 / 752.3 | 956.8 / 1,071.7 | 115.9 / 169.6 | 118.5 / 167.1 |
| 16 KiB | 701.8 / 792.3 | 936.0 / 1,044.9 | 153.8 / 190.3 | 116.7 / 150.6 |
| 64 KiB | 682.0 / 786.6 | 925.1 / 1,091.9 | 115.3 / 177.4 | 114.2 / 139.0 |
| 256 KiB | 1,076.0 / 1,139.8 | 937.4 / 1,054.9 | 114.6 / 136.5 | 117.1 / 129.8 |
| 1 MiB | 13,005.8 / 13,638.0 | 935.6 / 1,050.5 | 117.3 / 130.2 | 120.9 / 133.2 |
| 4 MiB | 15,374.4 / 16,209.4 | 915.6 / 1,049.2 | 113.4 / 140.0 | 133.7 / 155.0 |
| 16 MiB | 14,786.7 / 15,270.8 | 727.6 / 805.7 | 41.6 / 79.6 | 113.3 / 124.8 |
Additional notes
You may also notice that Inter CPU, i.e. ordinary Fast DDS message passing, shows a significant latency increase around 1 MiB.
To investigate this, I compared the latency distributions of Inter CPU and Inter SHM. One notable difference is that the Inter CPU latency has a much wider, bimodal distribution.
I have not yet investigated the exact cause in detail, but it may be related to the DDS configuration. For example, fragmentation of the DDS payload may occur for larger messages. If so, the behavior might be improved or stabilized by tuning the Fast DDS communication parameters.
In contrast, Inter SHM is much more stable. I believe this is because only a small descriptor is sent through DDS, rather than the actual payload. Since the descriptor is small enough to avoid the behavior seen with large DDS payloads, the pub/sub latency remains stable regardless of the actual payload size.
What’s next?
The implementation is almost complete, so I do not currently have a specific roadmap for this plugin.
However, if you try it and find any issues, I would be very happy to investigate them. Feedback is always welcome.