Why we did not extend /diagnostics?

ROS ROS General
,
1

Can /diagnostics carry a fault, or only a status?
We spent a year building a diagnostic layer for ROS 2 (ros2_medkit, earlier thread: What's missing from ROS 2 diagnostics (and what we built) ) and the question we get most (including from a ROSCon reviewer) is why we did not extend diagnostic_msgs, diagnostic_updater and the aggregator instead.
Here is the reasoning with the message definitions side by side, so the specifics can be argued with.

TL;DR: a status is information, a fault is a model you can query and act on, and the second does not fall out of the first by adding fields or more context.

What /diagnostics is, and does well

REP 107 (Tully Foote, November 2010) wrote down the diagnostics system Willow Garage had already built: a /diagnostics topic carrying DiagnosticArray of DiagnosticStatus, and an aggregator with analyzers. It names three things the system is for:

  • a quick look at normal operation
  • detailed debug information
  • and long-term logs for history.

Its best practices are robot_monitor on screen while the robot runs, and a rosbag record of /diagnostics in the bringup launch. The REP defined three levels, OK, Warn and Error. The message today:

byte OK=0
byte WARN=1
byte ERROR=2
byte STALE=3
byte level
string name          # "a description of the test/component reporting"
string message       # "a description of the status"
string hardware_id   # "a hardware unique string"
KeyValue[] values

diagnostic_updater publishes on a period (1s by default)

diagnostic_aggregator groups statuses by name rules (prefix, substring, regex) into /diagnostics_agg, publishes the worst level on /diagnostics_toplevel_state, and marks an item STALE when it stops updating within the analyzer timeout.

rqt_robot_monitor keeps the last 30 arrays in memory.

diagnostic_remote_logging sends statuses to InfluxDB.

diagnostic_analysis, the ROS 1 tool for recorded diagnostics, is “Not ported to ROS2 yet” in the ros2 README.

That is a live status stream. It works well for a person watching a screen, and we did NOT change any of it.

Where a status stream is not enough

Identity. A status is keyed by who reports it (name, “LiDAR scan0”). It can say what broke in message, hardware_id or a key-value, and people use those fields that way, but that is text. Nothing in the stack reads it as a fault identity. The aggregator keeps one status per name, the latest, and overwrites it with the next one. So the stack has no thing called “the fault”. I mean nothing knows when it started, how many times it came back, whether it is still active, or whether anyone acknowledged it.

rqt keeps the last 30 arrays, so a blip at night is gone by morning. People have asked for this before. In January 2018 someone on ROS Answers wanted a latched log of warnings and errors (“high level system log, latching diagnostic warnings/error”, question 280273). Nobody answered, and the asker wrote that they would probably build their own collector node.

Address. Say you have the rosbag from the three seconds around the failure. Where do you put it? hardware_id is just a free text. The aggregator tree is for display in rqt and a tag for the InfluxDB exporter, not a place where you can attach a recording.

Also, much of what breaks in a robot is not a ROS node: the MCU in a motor driver, a sensor with its own firmware. These appear in /diagnostics only if the driver author wrote a diagnostic task for them, and when the driver goes quiet the aggregator can only say STALE.

Operations and state. You cannot ask the stream about a fault. Is it still active? acknowledge it, change the configuration behind it, give me the recording. What the aggregator holds in RAM dies with the process. A bag of /diagnostics or the InfluxDB export survives a restart, but those are stored samples, not state. To know what is active now, you read the samples and work it out yourself. Whatever answers those questions has to be a separate service beside the ROS graph, one that holds fault records with an address and verbs. At that point you are no longer extending a message. Extending the aggregator ends in the same place. The aggregator does one thing: statuses in, one combined status out, with a fast path for watchdogs. Add fault ids, a database and an API and it is no longer an aggregator. It is a fault manager.

What we built beside it

ros2_medkit_msgs/Fault:

string fault_code               # e.g. MOTOR_OVERHEAT
uint8 severity                  # INFO WARN ERROR CRITICAL
string description              # the third argument of report()
builtin_interfaces/Time first_occurred
builtin_interfaces/Time last_occurred
builtin_interfaces/Time last_passed
uint32 occurrence_count         # counted on edges: a cleared fault that comes back adds one
string status                   # PREFAILED PREPASSED CONFIRMED HEALED CLEARED
string[] reporting_sources

The lifecycle is an AUTOSAR DEM-style debounce counter. FAILED events move it towards CONFIRMED, PASSED events move it towards HEALED. HEALED means the counter reached the healing threshold. CLEARED means a human acknowledged the fault through the clear service. Both kinds of record stay in the database.

Defaults worth knowing before you compare
  • Bare fault manager: confirmation_threshold: -1 (the first FAILED report confirms, set -N to require N) and healing_enabled: false. The bringup launch turns healing on with healing_threshold: 3 and switches black-box recording on.
  • CRITICAL skips a configured confirmation threshold.
  • FaultReporter (one include, report(code, severity, description)) has a client-side filter: a WARN is held until three reports in ten seconds, ERROR goes through at once, and PASSED is held the same way. The bridge reports through it, so three OK samples reach the manager as one PASSED.
  • What survives a clear: the compact freeze frame does. Ordinary snapshot rows and the default single rosbag per fault code do not, unless you set snapshots.retain_on_clear and raise the bag retention.
  • Freeze frame (entity-default capture): up to 16 topics published by the reporting ROS node, sampled after confirmation, /rosout and /parameter_events excluded, best effort.
  • Rosbag: our own RAM ring buffer, written through the rosbag2 writer to MCAP, 5 s before and 1 s after by default. It records the entity’s topics plus /tf when the source maps to a live node, otherwise the full buffer.
  • Storage: SQLite at /var/lib/ros2_medkit/faults.db. Faults, freeze frames and the near-miss series (FAILED reports that moved the counter without confirming) are still there after a restart.

On CONFIRMED the fault manager takes a freeze frame of the reporting node’s topics and with recording on, it writes its ring buffer to an MCAP bag. Each fault is linked by its reporting sources to an entity in a SOVD (ISO 17978) tree: Areas, Components, Apps, Functions. A function carries its own /faults, so “obstacle avoidance is degraded” maps to the app that is containing the fault. A device that never speaks DDS becomes an external component from the manifest, and a protocol plugin (OPC UA in the open repo) turns its alarms into faults on it. All of it is served over REST: list and stream faults (SSE), read one, clear it, read the entity’s data, PUT a configuration, download the bag from bulk-data, etc. A laptop, a fleet server or a tool can do that without joining the DDS graph.

And /diagnostics stays an input

ros2_medkit_diagnostic_bridge subscribes to /diagnostics and feeds the fault manager:

DiagnosticStatus level Fault event
OK PASSED, moves the fault towards HEALED when healing is enabled
WARN fault, severity WARN
ERROR fault, severity ERROR
STALE fault, severity CRITICAL (see caveats)

An existing diagnostic_updater gains a fault code, a lifecycle and a history. The node does not change.

Bridge details and caveats
  • Codes: taken from the name (motor: Temperature becomes MOTOR_TEMPERATURE) or mapped by hand.
  • Who owns the fault: by default the bridge itself is the reporting source, so a bridged fault hangs on the bridge node and gets no entity freeze frame of its own. Set use_hardware_id_as_source_id and put the driver’s full node name (with slashes) in hardware_id, or configure snapshots.default_topics. The log and action bridges point at the real node.
  • STALE is produced by the aggregator, so the STALE row only fires when the bridge listens to /diagnostics_agg.
  • Action servers: ros2_medkit_action_status_bridge turns an action with an ABORTED goal into one fault, healed on the next success, CANCELED ignored unless configured. /rosout comes in through ros2_medkit_log_bridge.
How the existing tools fit

diagnostic_updater and the aggregator stay the live status stream. We read it as input and change nothing. Grafana and the Canonical observability stack plot whatever you feed them, host metrics or /diagnostics levels through diagnostic_remote_logging. A dashboard plots samples. The fault, its owner, its evidence and the next action still need to live somewhere else. rosbag2 is the writer and the format we record into. Its snapshot mode has the same “buffer then write” shape, but its only trigger is a manual service call, so the fault manager keeps its own ring buffer and writes it when a fault confirms.

The question to diagnostics users

Where should the line be? Our answer: DiagnosticStatus stays the health stream, and the fault record, its entity and the REST API get agreed on separately. That is what we built. Should the fault record become a shared ROS message one day? That is the question for this group. The ROSCon talk goes through the trade-offs.
Repo: ros2_medkit

Sources
1 Like