Rethinking Rolling Syncs

Lyrical Luth has shipped and Rolling is back to being the active development target for M Turtle (May 2027) and the cycles after it. We want to change how packages move from Rolling into the main distribution mirror.

Current process

Getting a package from the build farm’s testing repository into the main apt archive requires someone on the release team to manually trigger a sync. A few consequences of that:

  • Syncs happen in batches, whenever someone has time to run and announce them.
  • A single regression in a candidate batch can hold back every other package in that batch, including ones that are fine.
  • Timing is unpredictable. A maintainer releases a package, then waits for someone to decide it’s sync day.

This process works but doesn’t scale well, and it ties release velocity to release team bandwidth instead of the actual state of the packages.

Alternative we considered and rejected: no sync to main

One option was to stop syncing Rolling to main altogether and leave Rolling packages on the testing mirror permanently. That removes the manual-sync problem entirely, since there’s nothing to schedule.

We’re not doing this because it breaks co-installation. Kilted and Lyrical live on the main mirror. If Rolling only lived on testing, there would be no supported way to install Rolling packages alongside a Kilted or Lyrical install on the same system. That mixed setup is a real workflow for people testing a Rolling-only package on top of a stable base, so we want to keep it working.

What we’re proposing

Sync becomes automatic once a candidate batch of packages clears a readiness threshold, instead of a person deciding sync day. Two factors will determine that threshold:

  1. Batch size. A larger, well-tested batch of ready packages clears a lower bar than a single isolated package change.
  2. Position in the dependency graph. A leaf package with few or no reverse dependencies is lower risk to sync than a package that hundreds of other packages depend on. Packages higher up the dependency graph need more validation and longer soak time before an automatic sync fires.

Most day-to-day package updates should reach main faster and on a more predictable schedule than today. Packages with wide downstream impact still get more scrutiny than they do under the current one-size-fits-all manual gate.

The threshold changes over time

The threshold is not fixed. As we approach the M Turtle release (May 2027), we plan to raise it: larger validation batches, stricter dependency-graph requirements, similar to how Rolling already gets more conservative ahead of a stable release under REP 2002. After M Turtle ships and Rolling resets for the next cycle, the threshold comes back down.

Rough sequence:

  • Now through early next year: relaxed thresholds, frequent automatic syncs.
  • Approaching the M Turtle release: thresholds tighten to protect the stabilization period.
  • After M Turtle ships: thresholds relax again for the next cycle.

We’ll publish the actual numeric thresholds and any changes to them on the release schedule ahead of time.

What isn’t changing

  • Stable distributions (Humble, Jazzy, Kilted) are unaffected. This is scoped entirely to Rolling.
  • The release process for maintainers is unaffected. You still release into Rolling via bloom the same way you do today. What changes is what happens after your package lands in the testing repo, not how it gets there.

Open questions

  • What’s the right starting batch size and dependency-graph cutoff? We have starting numbers in mind but want feedback before locking them in.
  • Should there be a manual override or expedite path for urgent fixes that shouldn’t have to wait for a batch to fill?

FAQ

Why not just leave Rolling on the testing mirror and skip main syncs entirely? It breaks co-installation. You’d lose the ability to install Rolling packages alongside a Kilted or Lyrical install on the same system. See the section above.

Does this affect Humble, Jazzy, or Kilted? No. This only changes how packages move from Rolling to main. Stable distributions keep their existing patch release process.

How is “readiness” measured? Build farm test results and each package’s position in the dependency graph. We’re not introducing a new quality metric.

Who sets the threshold, and can I see it? The release team owns the thresholds. We’ll publish them, and any changes to them, on the release schedule rather than adjusting them without notice.

What happens if a bad package gets synced automatically? Automation changes when a sync happens, not whether we can pull a package back out of main after the fact. That stays the same.

Will this make Rolling less stable to build against? It should do the opposite. Packages with wide downstream impact get a higher bar than the current manual gate applies to everything, while low-risk leaf packages stop waiting on unrelated issues.

Can I opt my package out of automatic syncing? No. This isn’t specific to automation. If something your package depends on changes and syncs, your package needs to be rebuilt against it whether or not you’d have preferred to wait. Manual syncs work the same way today.

When does this start, and when does the threshold tighten for M Turtle? This will be up to the M-turtle ROS Boss and release team, but we anticipate January of next year.

6 Likes

I like the idea, as it will allow packages to be tested more quickly without requiring users to switch to the testing repository.

Will it be possible in the future to also automatically accept bloom release PRs to rolling, specifically those that only update a package? Currently, simple package version updates can take a while.

1 Like

We have also been discussing removing human-review for some category of rosdistro pull requests. In this case, “simple” pull requests of minor/patch version bumps seem like a fairly easy thing to allow automatic merges on. There isn’t a timeline on it currently, but I’ll open a tracking issue that you can follow.

2 Likes

As it turns out, I had even opened an issue for it Consider merging version bumps automatically · Issue #48439 · ros/rosdistro · GitHub

Will give this a higher prioritization now.

1 Like

What’s the status of this? Rolling is still missing a sync since the switch to Ubuntu 26

Which repo URL are you using? The non-testing repo does not seem to contain rolling anymore, at least on Ubuntu 26.04. You need to switch to the testing repo (sudo apt install ros2-testing-apt-source). I have not checked, if the packages there are synced automatically after a successful build.

The feature work is being tracked here: Auto-trigger sync-to-main after successful sync-to-testing by Yadunund · Pull Request #1151 · ros-infrastructure/ros_buildfarm · GitHub

1 Like

A side question to that: Rolling sync to testing has been broken quite a lot in recent months. What’s causing this? Is there a list of binary builds that have to succeed in order for this to run and if so, is it being worked on to fix this?