Posted by @ptientho:
Hi @aaronchongth, @0RBalaji, Yes the problem is within my fleet adapter that it somehow does not respond to RMF commands as in the following log message
[rmf_task_dispatcher-7] [ERROR] [1748960731.495016394] [rmf_dispatcher_node]: Dispatch command [0] type [1] for task [patrol.dispatch-1] directed at fleet [Fastbot] has expired. This likely means something is wrong with the fleet adapter for [Fastbot] preventing it from responding.
I also found warnings from Zenoh side, which is about Timeout 10s ! Not sure if the problem is related to that.
zenohd
2025-06-03T13:24:02.736322Z INFO main ThreadId(01) zenohd: zenohd v1.4.0 built with rustc 1.85.0 (4d91de4e4 2025-02-17)
2025-06-03T13:24:02.736636Z INFO main ThreadId(01) zenohd: Initial conf: {"access_control":{"default_permission":"deny","enabled":false,"policies":null,"rules":null,"subjects":null},"adminspace":{"enabled":true,"permissions":{"read":true,"write":false}},"aggregation":{"publishers":[],"subscribers":[]},"connect":{"endpoints":[],"exit_on_failure":null,"retry":null,"timeout_ms":null},"downsampling":[],"id":"44b1e5a093a1a1e6d46fe9c4590ae47","listen":{"endpoints":{"peer":["tcp/[::]:0"],"router":["tcp/[::]:7447"]},"exit_on_failure":null,"retry":null,"timeout_ms":null},"low_pass_filter":[],"metadata":null,"mode":"router","namespace":null,"open":{"return_conditions":{"connect_scouted":null,"declares":null}},"plugins":{},"plugins_loading":{"enabled":true,"search_dirs":[{"kind":"current_exe_parent","value":null},".","~/.zenoh/lib","/opt/homebrew/lib","/usr/local/lib","/usr/lib"]},"qos":{"network":[],"publication":[]},"queries_default_timeout":null,"routing":{"interests":{"timeout":null},"peer":{"linkstate":{"transport_weights":[]},"mode":null},"router":{"linkstate":{"transport_weights":[]},"peers_failover_brokering":null}},"scouting":{"delay":null,"gossip":{"autoconnect":null,"autoconnect_strategy":null,"enabled":null,"multihop":null,"target":null},"multicast":{"address":null,"autoconnect":null,"autoconnect_strategy":null,"enabled":true,"interface":null,"listen":null,"ttl":null},"timeout":null},"timestamping":{"drop_future_timestamp":null,"enabled":null},"transport":{"auth":{"pubkey":{"key_size":null,"known_keys_file":null,"private_key_file":null,"private_key_pem":null,"public_key_file":null,"public_key_pem":null},"usrpwd":{"dictionary_file":null,"password":null,"user":null}},"link":{"protocols":null,"rx":{"buffer_size":65535,"max_message_size":1073741824},"tcp":{"so_rcvbuf":null,"so_sndbuf":null},"tls":{"close_link_on_expiration":null,"connect_certificate":null,"connect_private_key":null,"enable_mtls":null,"listen_certificate":null,"listen_private_key":null,"root_ca_certificate":null,"so_rcvbuf":null,"so_sndbuf":null,"verify_name_on_connect":null},"tx":{"batch_size":65535,"keep_alive":4,"lease":10000,"queue":{"allocation":{"mode":"lazy"},"batching":{"enabled":true,"time_limit":1},"congestion_control":{"block":{"wait_before_close":5000000},"drop":{"max_wait_before_drop_fragments":50000,"wait_before_drop":1000}},"size":{"background":2,"control":2,"data":2,"data_high":2,"data_low":2,"interactive_high":2,"interactive_low":2,"real_time":2}},"sequence_number_resolution":"32bit","threads":1},"unixpipe":{"file_access_mask":null}},"multicast":{"compression":{"enabled":false},"join_interval":2500,"max_sessions":1000,"qos":{"enabled":false}},"shared_memory":{"enabled":true,"mode":"lazy"},"unicast":{"accept_pending":100,"accept_timeout":10000,"compression":{"enabled":false},"lowlatency":false,"max_links":1,"max_sessions":1000,"open_timeout":10000,"qos":{"enabled":true}}}}
2025-06-03T13:24:02.736911Z INFO main ThreadId(01) zenoh::net::runtime: Using ZID: 44b1e5a093a1a1e6d46fe9c4590ae47
2025-06-03T13:24:02.737739Z INFO main ThreadId(01) zenoh::net::runtime::orchestrator: Zenoh can be reached at: tcp/[fe80::42:c0ff:fea8:b008]:7447
2025-06-03T13:24:02.737759Z INFO main ThreadId(01) zenoh::net::runtime::orchestrator: Zenoh can be reached at: tcp/192.168.176.8:7447
2025-06-03T13:24:02.737787Z INFO main ThreadId(01) zenoh::net::runtime::orchestrator: zenohd listening scout messages on 224.0.0.224:7446
2025-06-03T13:26:04.694405Z WARN net-0 ThreadId(03) zenoh::net::routing::dispatcher::queries: Face{2, 8cb23f808ff07dc5dba172b3b7a043bd}:3 Didn't receive final reply for query Face{3, 989b7be71feb97cf8df6f7c62d5124f1}:3: Timeout(10s)!
2025-06-03T13:26:04.694700Z WARN rx-1 ThreadId(07) zenoh::net::routing::dispatcher::queries: Face{2, 8cb23f808ff07dc5dba172b3b7a043bd}:3 Route reply: Query not found!
2025-06-03T13:26:04.694727Z WARN rx-1 ThreadId(07) zenoh::net::routing::dispatcher::queries: Face{2, 8cb23f808ff07dc5dba172b3b7a043bd}:3 Route final reply: Query not found!
2025-06-03T13:26:21.704674Z WARN net-0 ThreadId(03) zenoh::net::routing::dispatcher::queries: Face{2, 8cb23f808ff07dc5dba172b3b7a043bd}:4 Route reply: Query not found!
2025-06-03T13:26:38.712272Z WARN net-0 ThreadId(03) zenoh::net::routing::dispatcher::queries: Face{2, 8cb23f808ff07dc5dba172b3b7a043bd}:5 Didn't receive final reply for query Face{3, 989b7be71feb97cf8df6f7c62d5124f1}:5: Timeout(10s)!
2025-06-03T13:26:38.713500Z WARN rx-0 ThreadId(06) zenoh::net::routing::dispatcher::queries: Face{2, 8cb23f808ff07dc5dba172b3b7a043bd}:5 Route reply: Query not found!
2025-06-03T13:26:38.713547Z WARN rx-0 ThreadId(06) zenoh::net::routing::dispatcher::queries: Face{2, 8cb23f808ff07dc5dba172b3b7a043bd}:5 Route final reply: Query not found!
2025-06-03T13:26:55.721681Z WARN rx-1 ThreadId(07) zenoh::net::routing::dispatcher::queries: Face{2, 8cb23f808ff07dc5dba172b3b7a043bd}:6 Route final reply: Query not found!
2025-06-03T13:26:55.721681Z WARN net-0 ThreadId(03) zenoh::net::routing::dispatcher::queries: Face{2, 8cb23f808ff07dc5dba172b3b7a043bd}:6 Didn't receive final reply for query Face{3, 989b7be71feb97cf8df6f7c62d5124f1}:6: Timeout(10s)!
2025-06-03T14:30:48.738590Z WARN net-0 ThreadId(03) zenoh::net::routing::dispatcher::queries: Face{2, 8cb23f808ff07dc5dba172b3b7a043bd}:655 Didn't receive final reply for query Face{5, 9d4dc4783e3ada48578489573a036e61}:69: Timeout(10s)!
2025-06-03T14:30:48.739779Z WARN rx-1 ThreadId(07) zenoh::net::routing::dispatcher::queries: Face{2, 8cb23f808ff07dc5dba172b3b7a043bd}:655 Route final reply: Query not found!