Issue with Gazebo not launching after defining Mutex Groups in Traffic Editor

Hi @Yadunund @Aaron_Chong
I have been using Open RMF by source installation, and till now I was using a package of traffic editor in which the mutex property in lane and waypoints was not defined

Now, as I have shifted to the new traffic editor package this one package in which we have mutex property ,so while launching my launch file, I have been facing an issue; my Gazebo is not being launched
These are the logs. Only this much log is coming, and then nothing comes after this
https://gist.github.com/mechdrive/a0ec91d32b6e52f74e5d2db113565b90

Looking at your logs this may be relevant.

I don’t think this is the cause of the issue. see the logs of my another workspace there Gazebo launched properly . It seems the problem might be specific to my current setup rather than the Server.emit() call itself.

To confirm, has the mutex property for lanes and waypoints been released in the ROS 2 Humble version? I have been trying to use this property.

As I understand, to use a mutex group, we need to define the property in the waypoints and lanes, and include the following in common.launch.xml:
<!-- Mutex Group Supervisor: manages the locking and unlocking of mutex groups --> <group> <node pkg="rmf_fleet_adapter" exec="mutex_group_supervisor"> <param name="use_sim_time" value="$(var use_sim_time)"/> </node> </group>l
Am I correct? Apart from this, is there anything else that needs to be included to enable mutex group functionality?

So you will need to build main from source for mutex groups to work correctly for humble. I don’t think humble has mutex groups enabled by default in the binary.

I’m using source build only not binary

Mutex groups have no effect on the Gazebo simulation. All logic related to mutex groups is handled by various ROS nodes that live outside of Gazebo. None of the Gazebo plugins for Open-RMF are even aware of the concept of mutex groups.

Looking at your log, one thing that stands out to me is that gzserver stopped logging anything during the model download phase. If I were to take a guess, it may be struggling to download one of your models. Perhaps the model is very large or its url is broken. The log doesn’t show Gazebo crashing, so when you say it’s “not being launched”, I assume you mean it’s stalling out rather than exiting / crashing.

Thanks for clarifying that mutex groups don’t affect Gazebo directly. I’d still like to understand how to correctly use them in Humble with a source build.

  • What are the exact steps needed to enable and configure mutex groups?
  • Beyond defining the mutex property in waypoints/lanes in the traffic editor, I have also launch the mutex_group_supervisor node in my launch file (common.launch.xml),

and I have this file for the mutex group supervisor
‘src/rmf/rmf_ros2/rmf_fleet_adapter/src/mutex_group_supervisor/main.cpp’
main.cpp

It sounds like you’re already doing everything necessary to use the mutex group feature.

Once you’ve figured out why gazebo isn’t launching for you, you can test whether the mutex groups are working the way you expected and then follow up with any further questions about using them.

My Gazebo setup is working fine in my previous workspace where I had all the packages from the Humble branch. The issue with Gazebo not launching only started after I updated the rmf_traffic_editor package in the current workspace.

So for now, I will continue working in the previous workspace. In that case, to use the mutex group feature, what additional updates or changes do I need to make?

It’s important to make sure you’re using matching branches for rmf_traffic_editor, rmf_simulation, and rmf_demos since changes that happen to gazebo will require updates across all of those packages.

The main branches of the core Open-RMF libraries are kept compatible with humble since we know there are users who have deployed RMF systems on top of humble, but it’s generally not feasible to support old versions of gazebo alongside new versions due to API and behavior changes in gazebo. This incompatibility is likely to be the reason gazebo isn’t launching for you. I don’t think we support “gazebo classic” at all beyond the humble distro of the traffic editor, so you’ll need to update to gz-sim.

If you need to simulate the latest features of Open-RMF, I recommend at least using jazzy for a simulation machine. You can use docker to run a jazzy simulation inside of an Ubuntu 22.04 machine.

A few notes on this that I believe are the real cause of what you are seeing.

We recently changed the model download behavior in traffic editor to point at models in Fuel instead of downloading models at build time. We have not actually tested this with Gazebo Classic that you seem to be using because Gazebo classic has been EOL since 8 months ago, as shown in your log:

[gzserver-19] This version of Gazebo, now called Gazebo classic, reaches end-of-life
[gzserver-19] in January 2025. Users are highly encouraged to migrate to the new Gazebo
[gzserver-19] using our migration guides (https://gazebosim.org/docs/latest/gazebo_classic_migration?utm_source=gazebo_ros_pkgs&utm_medium=cli)

Now Gazebo Classic should still be able to download models from fuel. The logs show that it was able to download the Coke mesh:

[gzserver-19] [Msg] Downloaded model URL: 
[gzserver-19]           https://fuel.gazebosim.org/1.0/OpenRobotics/models/Coke
[gzserver-19]       to: 
[gzserver-19]           /home/poonam/.ignition/fuel/fuel.gazebosim.org/openrobotics/models/coke/tip

But still, I think my recommendation would be to change from Gazebo classic to Gazebo by changing your launch command:

# Before
ros2 launch rmf_demos_gz_classic office.launch.xml
# After
ros2 launch rmf_demos_gz office.launch.xml

Since, again, Gazebo classic has been EOL for some time already.

Thanks a lot for the guidance!
I’ve now shifted my setup to ROS 2 Jazzy with gz-sim, and everything is working properly.

@grey
I do have one clarification regarding the mutex group behavior:
If a robot has completed its assigned task and currently has no active task IDs,

i’m getting this log

Blockquotefleet_adapter-17] [ERROR] [1759140882.224763965] [tinyRobot_fleet_adapter]: Forcibly releasing mutex group [m1] requested by robot [tinyRobot/rgv5] because the robot has been idle for an excessive amount of time.

As you can see in my setup, the robot rgv3 completed its task while inside a lane that belongs to a mutex group. I’ve set the finishing request to "nothing", so after completion it just stays there.

  • In this case, since rgv3 is still holding that position, will it continue to hold the mutex group lock even though it has no active task assigned?
  • If another robot receives a new task that requires passing through the same waypoint in that mutex group, will rgv3 move to clear the path for that robot as this rgv3 is not having any task id

this is happening when robot is already with active task id

In general it’s recommended to always have a finishing task (typically parking or charging). If your robot is left in a completely idle state for a prolonged period of time, then it can be very disruptive towards the overall traffic system. If there are any traffic conflicts between the idle robot and any other robots, the traffic will get jammed up.

You could turn on “responsive_wait” which will kick in if your robot is idle and has no finishing task. We don’t generally recommend using the responsive wait feature because it can still be somewhat disruptive for overall traffic, but the feature exists in case there are users that have a need for it.

The responsive wait behavior will hold onto the mutex, which seems to be what you want. However, you should keep in mind that because of the way mutexes work, that mutex will never be released by your idle robot until it gets a new task and leaves the mutex area. So if your mutex group covers multiple lanes or vertices, then all those lanes and vertices will be locked down for as long as your robot is idle.

In general what we recommend for the cleanest overall behaviors is

  1. Always give your robots finishing tasks to go park or charge somewhere that’s out of the way
  2. The parking and charging spots should not be part of a mutex group that includes any other lanes or vertices. Usually it does not have a mutex group at all.

If I set the finishing request to “parking”, I’d like to clarify a few behavioral points for my warehouse layout:

  1. In my setup, robots are not required to return to their parking positions after every task. So, if a robot completes a delivery and starts moving toward its parking position, what happens if a new task becomes available during that time?
  • Will the robot cancel or interrupt the parking task to take up the new task immediately?
  • Or will it first complete the parking task and then begin executing the newly assigned task?
  1. Additionally, if I have already assigned 2 tasks to the same robot, does RMF’s task bidding system take into account the travel cost to the parking position when evaluating or scheduling the second task?

Whenever a new task request is assigned to the robot (and scheduled to begin immediately), the robot will automatically and immediately cancel its “finishing task” to start performing the new request. So in the case you described, the robot would immediately start doing the newly available task request.

Will the robot cancel or interrupt the parking task to take up the new task immediately?

Yes, this.

Additionally, if I have already assigned 2 tasks to the same robot, does RMF’s task bidding system take into account the travel cost to the parking position when evaluating or scheduling the second task?

I don’t think the finishing task is accounted for during task planning, but I can’t think of a situation where it would affect the outcome of the plan even if it was accounted for. The task planner does account for the current position of the robot, so if the robot is being assigned a task while it’s parked, then the parking spot will be taken into account. If your robot is not currently near its parking location then there’s not much value in factoring it into the task plan (although I’m open to discussing this further if anyone disagrees).

The task planner also does account for the robot’s charging location, which is usually the same as the parking location (or at least nearby), so the tasks assigned to robots are likely to be grouped near the charging location when possible.