In the production-scale ROS systems I’ve been involved with we have mostly ignored the ROS logging system as much as possible. Instead we redirect our logs to cloud logging providers like Mezmo or Betterstack.
The reason for this is that being able to log in JSON format has been invaluable, and it allows us to include debugging and metadata information beyond a simple log string.
Cloud logging providers make it easy to gather statistics, create errors for alerts and review history. We can also share links to log lines and collect time stamps, filter by system and process, it’s very handy.
For our own nodes and processes we use rich logging libraries to log with metadata directly to a logging agent.
For ROS nodes we don’t write, we use the RCUTILS_CONSOLE_OUTPUT_FORMAT to format to an intermediate format which then gets translated to Json by our logging agent (we use fluentbit currently)* And sent to the cloud.
The one downside to this is we don’t have rich logs in our rosbags, which is frustrating, but we have found the tools for reviewing logs from bags so inferior to the web services it just doesn’t matter.
* We did this because we ran into issues having RCUTILS_CONSOLE_OUTPUT_FORMAT log Json directly, but I see Achille above got it working, I’ll have to review what issue we ran into.