How many DDS participants are currently used/allowed by RMW?

ROS ROS General
,
1

We’ve currently been running into some issues that look like reaching the maximum number of DDS entities (i.e. nodes are started without error, but do not appear in ros2 node list, neither do their topics appear). If we shut down a few other nodes, than the same node runs well.

Overflowing into other domains should not be an issue according to The ROS_DOMAIN_ID — ROS 2 Documentation: Galactic documentation (we don’t use any other domains than 0). We don’t yet have 120 nodes, but I guess we’ll get there.

However, with the changes from Switch to one participant per context by ivanpauno · Pull Request #189 · ros2/rmw · GitHub , I got a bit confused about the number of participants created and needed by ROS 2.

Now it should be the case that there is 1 DDS participant per Context. However, the explanation of what Context currently is is a bit vague.

Is it the case that the simple single-node process creates exactly 1 participant?

How is it with component container? Does it have 1 participant for the whole container, or does it have one per loaded component?

How is it with the weird transform_listener_impl_* nodes automatically created by TransformListener? Do they “eat” a participant?

And last - I guess the 120 participant limit is per PC, right? This isn’t directly mentioned in the design doc, but I think it follows from the design…

To resolve our current problem, I’ll try switching to discovery server, but I’d like to have a clear view on the other mentioned questions.

Thanks for ideas!

1 Like
2

A context in this … erm… context is an rmw_context_t. These are plumbed all the way out to the client libraries. So an rclcpp::Context has an rcl_context_t which has an rmw_context_t.

Related: When you call rclcpp::init(), you initialize the “global default context”. Many APIs will use the global default context when you don’t specify a context as a convenience.

A single node process would create one context, which means 1 participant for the DDS based RMW implementations.

A component container creates 1 context, meaning it also has 1 participant for the whole container.

2 Likes
3

We have seen the exact behavior you are describing when using fastDDS (Nodes not being listed / services not being found etc). We simply switched to cyclone and it was gone.
This is our cyclone config, perhaps it’s useful to you :

RMW_IMPLEMENTATION: "rmw_cyclonedds_cpp"
CYCLONEDDS_URI: 
  <General>
    <AllowMulticast>true</AllowMulticast>
    <Interfaces><NetworkInterface name="lo"/></Interfaces>
  </General>
  <Discovery>
    <ParticipantIndex>auto</ParticipantIndex>
    <MaxAutoParticipantIndex>1000</MaxAutoParticipantIndex>
  </Discovery>
1 Like
4

Also, if you set ROS_AUTOMATIC_DISCOVERY_RANGE=LOCALHOST, you’re implicitly limited to 32 nodes by the rmw layer. This is independently true of both the FastDDS and CycloneDDS rmw layers, and IMO is a pretty severe undocumented landmine. With CycloneDDS, you at least get loud complaints on the console when you hit the limit, but FastDDS just silently stops working in the manner you describe. Both limits can be overridden if you write your own DDS XML configuration, but to configure them you first have to know that they’re there.

1 Like
5

OMG DDS == OMG WTF? I can’t believe such settings are the default in ROS 2… @Jaime_Martin_Losa ?

But the 32 participant limit would better reflect what I saw on our systems…

If it’s too large then we will send a lot of announcement traffic.

On LOCALHOST, right? :smiley:

6

I looked a the code, and the 32 in the case of localhost is actually an improvement over the default of 4 in the non localhost version.

I am actually not sure what this parameter really does. The description here 10.7. Transport descriptors - 3.3.0 is also not really helping with that…

But it can’t be the maximum number of connectable nodes, as in this case no default setup with more than 4 nodes would work.