Buildfarmer Dashboard: ROS & Gazebo test health in one place

Infrastructure Project Infrastructure General
, ,
1

We are pleased to announce that the Buildfarmer Dashboard is publicly available at:

What is the Buildfarmer Dashboard?

The dashboard is a public, read-only view into the health of the ROS and Gazebo CI regressions, created and maintained by buildfarmers (@miguelgonrod, @Crola1702) , a subteam of the OSRF Infrastructure PMC. Until now, getting answers to common CI questions (why is this test failing? is this a known issue? is this platform-specific?) required reaching out to a buildfarmer directly or searching in the repositories issues.

The dashboard changes that. It gives developers, contributors, and PMC members a self-service way to answer those questions, including:

  • Active test regressions: consistent failures with the date they first appeared, affected platforms (OS/architecture), and a link to the relevant GitHub issue when one exists
  • Flaky tests: tests with intermittent failures over a rolling window, with failure rate and cross-platform signatures
  • Suspect commits: a direct GitHub compare link between the last successful build and the first failing build, so you can see exactly what changed (not always present due to builds cleaned)
  • Known issues: the current triage status of tracked failures, including whether they are under investigation or going stale (this status is reported/defined manually by the buildfarmers)

We want your feedback

This is the first public release, and we are actively looking for input from the community, especially from people who work with ROS/Gazebo CI regularly and from newcomers who are trying to understand the infrastructure for the first time.

Specifically, we would love to hear:

  1. Navigability: is it clear where to look for the information you need?
  2. Usefulness: does the data shown actually help you triage a failure?
  3. Missing context: is there something you routinely need that is not surfaced here?
  4. Any other suggestions: UI, performance, terminology, anything at all

Where to put your feedback?

Feel free to write your findings, ideas and suggestions in this discourse post!

8 Likes
[Announcement] The Buildfarm Dashboard is Live – Feedback Wanted!
2

Great work guys! And windows needs some love it seems (but that we know)

First piece of feedback, I was expecting that I would be able to click on the test name and it would either forward me to the Jenkins build part that failed or showed more details about the failed test? Or at least the latest one (since this is also counting all the tests)

3

This is great work indeed!

I’d like to pass on some feedback I got. It seems like there are additional flaky tests that have been flagged by the Gazebo team but don’t show up in the dashboard. Here are some examples:

It might be that some of the issues that show up when searching for flaky might have already been fixed, but the two issues I listed above are have not been resolved yet.

If it’s a matter of adjusting thresholds in the script analyzing the test results, maybe we can try relaxing them so that these flaky tests show up on the dashboard sooner.