We are pleased to announce that the Buildfarmer Dashboard is publicly available at:
What is the Buildfarmer Dashboard?
The dashboard is a public, read-only view into the health of the ROS and Gazebo CI regressions, created and maintained by buildfarmers (@miguelgonrod, @Crola1702) , a subteam of the OSRF Infrastructure PMC. Until now, getting answers to common CI questions (why is this test failing? is this a known issue? is this platform-specific?) required reaching out to a buildfarmer directly or searching in the repositories issues.
The dashboard changes that. It gives developers, contributors, and PMC members a self-service way to answer those questions, including:
Active test regressions: consistent failures with the date they first appeared, affected platforms (OS/architecture), and a link to the relevant GitHub issue when one exists
Flaky tests: tests with intermittent failures over a rolling window, with failure rate and cross-platform signatures
Suspect commits: a direct GitHub compare link between the last successful build and the first failing build, so you can see exactly what changed (not always present due to builds cleaned)
Known issues: the current triage status of tracked failures, including whether they are under investigation or going stale (this status is reported/defined manually by the buildfarmers)
We want your feedback
This is the first public release, and we are actively looking for input from the community, especially from people who work with ROS/Gazebo CI regularly and from newcomers who are trying to understand the infrastructure for the first time.
Specifically, we would love to hear:
Navigability: is it clear where to look for the information you need?
Usefulness: does the data shown actually help you triage a failure?
Missing context: is there something you routinely need that is not surfaced here?
Any other suggestions: UI, performance, terminology, anything at all
Where to put your feedback?
Feel free to write your findings, ideas and suggestions in this discourse post!
Great work guys! And windows needs some love it seems (but that we know)
First piece of feedback, I was expecting that I would be able to click on the test name and it would either forward me to the Jenkins build part that failed or showed more details about the failed test? Or at least the latest one (since this is also counting all the tests)
I’d like to pass on some feedback I got. It seems like there are additional flaky tests that have been flagged by the Gazebo team but don’t show up in the dashboard. Here are some examples:
It might be that some of the issues that show up when searching for flaky might have already been fixed, but the two issues I listed above are have not been resolved yet.
If it’s a matter of adjusting thresholds in the script analyzing the test results, maybe we can try relaxing them so that these flaky tests show up on the dashboard sooner.
Hi Kimberly, thanks for the suggestion! We already include a link to the corresponding Jenkins job in the Platform column, but I agree it would be much more convenient if clicking on the test name took you directly to the relevant test details or the latest failed run. That’s a great idea, I’ll open a PR to add that.
Thanks, Addisu! Let me take a closer look at those cases.
From a quick check, the PeerTracker.PeerTrackerStale test does appear on the Known Issues page. The ApplyLinkWrenchTestFixture.PersistentFromTopic case seems to be different, I haven’t seen it in the daily reports either, so it may be an issue with the backend of the reporting workflow. I’ll investigate the full pipeline to see why it’s not being picked up.
I also agree that relaxing the thresholds to surface flaky tests earlier is a good idea. I’ll look into adjusting them and see what changes make sense.
it is perhaps a little difficult to see which of the regressions (first tab) already have a ticket attached (second tab). Would it be possible to link these somehow, albeit for people to do that manually or something?
Is it possible to add some form of graph of the regressions compared to the build number or type? Perhaps just based on the nightlies perhaps like this from this windows ci job:
Regarding your first point, in theory this is already possible through the Known issue column in the dashboard. On the second page of the Known issues tab you’ll find all currently open known issues, even if the corresponding failure did not occur in today’s build. The first page only shows known issues that were observed in the latest run.
If a failure doesn’t have a linked known issue but there is already a GitHub issue for it, it is usually because we didn’t open that issue ourselves. In those cases we may not have been tracking it yet in our database, so we really appreciate it when people let us know, then we can then link the existing issue to the corresponding failure. Was this what you were referring to?
Regarding your second point, you can currently get a historical view by clicking 90d in the History column. That view shows, for each build, whether the test passed, failed, or didn’t run (for example because it was skipped or the job was cancelled), although it is visually more compact than the Jenkins history view.
Were you referring to that feature, or were you suggesting changing the visualization to look more like the Jenkins graph?
Finally, about the source code: the project is split into two parts. The dashboard itself is public and can be found here:
The data generator, which queries the build farms directly, is kept in the private buildfarm-tools-private repository. I’m not sure whether you have access to that repository, but if you do, that’s where you’ll find the implementation.
Thanks! I had indeed missed the known issue column as it was already empty. And I didn’t realize the 90d button could show that graph
About the accumulated error graph, I was referring to perhaps some kind of 90 day top view of the amount of errors per platform (so not per error), for the sake of an overview view. So in that sense the jenkins type graph that I showed before is only to compliment the features you have there in now (adding up the errors there are per platform)
Not sure if that is easily doable though? but might be a good graph for people to look on what the general status is. But this is more of a feature request then something that is absolutely missing, so more a nice-to-have.
Otherwise, I’m already looking a lot at this dashboard so great work!
