Performance Characteristics: subscription callback signatures, RMW implementation, Intra-process communication (IPC)

[mini-1235]

Hi @JM_ROS @wjwwood, thanks for your reply. It’s been a pretty busy week for me, but I finally had some time to think through / test the suggestions you mentioned.

@wjwwood

When talking about this stuff, I’d recommend explicitly using the phrase “intra-process comms” rather than “IPC”, as everywhere else on the internet IPC means inter-process comms (see Inter-process communication - Wikipedia ), which is sure to be a source of confusion. If you must abbreviate it, use something like “intra-”, “intra-pc”, etc. In my experience, people constantly confuse inter-process and intra-process even in the context of ROS and because of that I always just type out intra-process and inter-process explicitly. It’s worth the effort imo.

Sure, that make sense. I am trying to update the post to replace “IPC” with “intra-process comms”, but I am unable to find an edit button. If editing is still possible, please let me know and I’ll be happy to update it

@wjwwood

How did you determine the number of copies? Just code review or did you instrument it somehow?

• I’m surprised by the copies with const & T callbacks, that should be no different from shared_ptr<const T>, as all we’d need to do is dereference the shared-const ptr before passing it as a const reference. Do you know why that’s the case?
• See rclcpp/rclcpp/include/rclcpp/any_subscription_callback.hpp at c85ff926d2ea6c3fb8b8075e28b93632a791fd5c · ros2/rclcpp · GitHub , it really seems like const & T would be no different from shared_ptr<const T>.

Inspired by the demo in ros2_documentation, I am printing the address in each subscription callback.

Take const std_msgs::msg::Int32 & msg as an example. It enters the condition here: rclcpp/rclcpp/include/rclcpp/any_subscription_callback.hpp at c85ff926d2ea6c3fb8b8075e28b93632a791fd5c · ros2/rclcpp · GitHub
Adding

std::holds_alternative<ConstRefROSMessageCallback>(callback_variant_) ||
std::holds_alternative<ConstRefWithInfoROSMessageCallback>(callback_variant_)

to use_take_shared_method() seems to match the behavior of shared_ptr<const T>.

@JM_ROS

@wjwwood pointed out that the test was performed using the nav2 stack. This leads to the question if the nav stack actually sends ‘big’ messages in a high frequency. Judging from the results this seems not to be the case.

To my knowledge, the largest messages in the Nav2 stack are the global and local costmaps. With the default settings, the global costmap is published at 1 Hz and the local costmap at 2 Hz, and their sizes are on the order of a few hundred kilobytes. We are also subscribing to sensor data from Gazebo. I checked those as well, they are published at around 10 Hz with message sizes on the order of tens of kilobytes.

@JM_ROS

Try using a big message (> 10 MB) then you should see a difference in CPU load.

@wjwwood

I would need to look again, but it may be that intra-process is entirely “extra” work on top of inter-process, i.e. if we never avoid sending messages to the rmw layer for inter-process comms (e.g. when there’s no non-intra-process subscribers), then the intra-process comms is just more work on top of inter-process comms which always occurs. Ideally, the rmw layers would use less resources when you publish to them with no inter-process subscriptions present, but I’m not 100% sure about that either (this could be impacted by what QoS setting you’re using too, e.g. durability transient-local requires the inter-process to do most of the work when publishing, even when there are currently no subscriptions, due to needing to keep stuff around for late joining subscriptions).

• Follow up, I looked at the code and there’s definitely still logic to avoid inter-process when publishing under certain conditions (no inter-process subscriptions and not using transient-local durability), but I never got a chance to properly look at this pull request that added support for transient-local durability in intra-process ( Fix transient local IPC publish by jefferyyjhsu · Pull Request #2708 · ros2/rclcpp · GitHub ), so that might be something to have a look at, if we’re getting weird performance from intra-process.

I tried writing a quick demo that publishes larger messages, and I can clearly see a non-trivial reduction in CPU usage. When switching the qos to transient local, I do not observe any difference between inter-process and intra-process communication. I think this also explains the results in setup B, since the costmaps topics are published with transient local.

@wjwwood

How did you collect the cpu utilization? Did you wait for steady-state of some kind and then average a window of time? Did you run the experiment multiple times for each case? The differences in CPU utilization are close enough that it’s not clear to me weather it’s within the noise of the measurements or not (no error bars). Is this on a soft/hard real-time OS or just an OS with a standard scheduler, etc.

Yes. I usually wait about 1–2 minutes for the system to reach a steady state, and then compute the average CPU utilization. When preparing the report, I also repeated each experiment multiple times, and on different days. The error bars should be approximately ±0.3 and they are conducted on an OS with a standard scheduler.

@wjwwood

Put another way, how have you convinced yourself that you’re measuring the intra-process vs inter-process setting or something else?

@wjwwood

Related to clarifying how you’re collecting CPU statistics, looking at call graphs or flame charts or something before and after intra-process is turned on would be interesting, because it may be possible that inter-process comms (and copying messages) are not a significant source of CPU utilization in the first place.

Finally, to help explain the behavior I observed in setup A and to move this discussion forward, I also collected some flamegraphs. I have uploaded the SVG files here for convenience: https://drive.google.com/drive/folders/1A75ddUFnMZlh8DB0X-_BsRWAM-ovSJ-i?usp=sharing

zenoh (Intra-process comms off)

zenoh (Intra-process comms on)

Differential flamegraphs (./diff off on)

From the flamegraph and perf top, it seems that it is related to rclcpp::experimental::IntraProcessManager::matches_any_publishers() (?)

cyclone (Intra-process comms off)

cyclone (Intra-process comms on)

Differential flamegraphs (./diff off on)

Similar to the zenoh results, this seems to point to the rclcpp::experimental::IntraProcessManager::matches_any_publishers() as well.

I hope this helps move the discussion forward. Looking forward to your thoughts!