Hi Steve,
I quickly read through the paper, it is very well-researched, thank you for sharing.
It seems to me that the main cause of the increase in memory usage is the use of a DDS-based RMW implementation. Shared memory-based RMW implementations were dismissed due to stability issues.
With these preconditions, the results were not very surprising (although it is good to have the exact numbers from the paper), it is a consequence of how DDS works and the need for serialization and deserialization instead of just passing pointers like when intra-process communication is used.
I agree that it is a good practice to develop all nodes as components, so component composition can be used without additional effort.
However, I am not sure I agree with the final recommendation of compositing many components into a single process. Doesn’t this practice reduce the reliability and fault tolerance of the system? E.g. if the whole Nav2 stack is composited into a single process, then a crash in any of those components will lead to the crash of the whole stack. This might be acceptable in some cases (like in the case of a very resource-constrained system) but may not in others.
What do you think? Do you see this as a risk of using composition?
For the long term, I would rather see more investment in alternative ROS2 backends where the issues outlined in the paper can be fixed:
- Do not use network protocol-based RMW backends for intra-host communication, but use protocols that work without serialization / deserialization. A shared memory-based / zero-copy transport can be a good fit. Besides the standard POSIX shared memory, I would also look at Android’s Binder framework as an implementation. We know that it has pretty good performance (since billions of phones use it for almost every IPC need), and also has nice security features (e.g. both identity and token-based authentication)
- For inter-host communication, a protocol like Zenoh (https://zenoh.io) could be adopted. In the Zenoh model, we can have a single router / gateway process on each host that is in charge of the inter-host communication and the translation from intra-host communication to a network protocol. The protocol itself has less overhead than DDS, and it was already evaluated by the ROS TSC.
Kind regards,
Gergely