Never miss a backport again — automatic ABI checking for ROS 2 core repos (prototype ready)

Hi ROS community! :waving_hand:

Have you ever debugged a problem on Humble or Jazzy for hours… only to discover the fix already existed on rolling, but was never backported? :weary_face: You’re not alone — and I’d like to fix this systematically.

:confounded_face: The pain

In principle, all bug fixes should be backported to the supported downstream distributions — precisely because they are fixes. Today, however, this depends entirely on the maintainer’s manual effort and judgment, so backports happen on a need-to-do or requested basis rather than automatically and systematically.
One of the key blockers: it’s not always easy to tell whether a fix is ABI compatible with released distributions just by looking at the source code. This depends on developer skill and experience.
And too often, community developers hit a bug on a released distro, dig all the way down to the root cause, and only then find the fix was already merged to rolling — so they have to request the backport themselves. That’s wasted effort for everyone. :hourglass_not_done:

Making this automatic would save significant time for maintainers, developers, and users alike. Original discussion: ros2/rclcpp#2555

:sparkles: The prototype (working today!)

I built this as two GitHub Actions:

:wrench: libabigail-action — generic, repo-agnostic. Diffs two shared libraries with libabigail’s abidiff. Useful for any C/C++ project, not just ROS!

:robot: ros2-abi-action — the ROS-aware layer. For every PR it:

  • builds the package twice (target branch + PR head) in the matching distro container :building_construction:
  • runs the binary ABI diff
  • applies REP-0009 policy automatically: strict :cross_mark: for released distros, advisory :warning: for rolling
  • surfaces results as a sticky PR comment, ABI compatible :white_check_mark: / ABI break :collision: labels, and a pass/fail check

Maintainers can decide on backports at a glance. And the per-repo integration is a ~10-line workflow file, so rclcpp, rcl, rcutils, rmw and friends can all share the same setup. :tada:

:warning: Small print: an ABI compatible label is information, not an obligation to backport — behavioral changes can still surprise users even when ABI is intact. And this is binary ABI only; header templates/inlines still need human eyes. :eyes:

:folded_hands: I’d love your feedback

  1. Does this approach make sense for ROS 2 core repos? Concerns about CI cost or noise? :money_with_wings:
  2. Tooling: libabigail vs. the older abi-compliance-checker (used by industrial_ci / auto-abi-checker) — experiences? :hammer_and_wrench:
  3. Should we go further and wire the ABI compatible label into the Mergify backport flow? :repeat_button:
  4. Would community packages benefit from this as a shared reusable workflow too? :globe_showing_europe_africa:

Comments, use cases, and contributions all very welcome! :rocket:

Tomoya

16 Likes

This looks like a fantastic contribution to the ecosystem. Responding to your questions below.

I think it absolutely makes sense. If the intention is to run this as a github action, that shouldn’t add to the CI cost, as the github actions on public/open source repos are free (at least for now). Even if this was part of Jenkins, I think the incremental CI cost is small compared to the savings that we can get in less CI churn around backports over time.

No experiences to dicuss either way here, but libabigail seems mature and stable and reasonable dependency to take in CI here.

Probably not a bad idea? Maybe something to experiment with in a few repos before turning on everywhere.

Absolutely. We aren’t strictly enforcing ABI/API when we are merging new rosdistro bumps, but instead rely on the author’s experience and judgement on if it makes sense for their package in that distribution. I think providing these sorts of tools allows people to get more signal about the downstream impacts of changes in their packages. It also allows downstream consumers to be more confident in depending on a package if they know that the author is being diligent about API/ABI

2 Likes

I think this could also be useful beyond bug fixes.

I’m not sure what the general policy is around backporting new features to older supported distributions, especially LTS releases, but there are definitely cases where this would be valuable if the change is ABI-compatible and low risk.

One example that comes to mind is rosbag2. We’re still using Jazzy because it’s an LTS release. I implemented a feature to record all latched/transient-local messages without having to list the topics one by one, only to discover, while preparing the PR for Rolling, that essentially the same feature was already available there.

We have a similar case with circular rosbag recording. Today we have another node handling this, while newer rosbag2 versions can provide a simpler solution directly.

We’re starting to think about testing Lyrical this year, but moving a production robotics stack to a new ROS distribution is obviously something we need to plan and prioritize carefully.

So from our side, it would be useful if this tooling could help identify not only bug fixes, but also small, self-contained, ABI-compatible features that could reasonably be backported to supported LTS releases. There are quite a few improvements in newer ROS 2 versions that users of an LTS distribution could benefit from without necessarily having to upgrade the entire stack.

There isn’t really a general policy for all ROS 2 packages. It is ultimately up to the owner/maintainer of each package to set their policy for what they would accept in a version of their package released to a distribution. We frequently see maintainers bumping minor and even major versions of a package with the addition of new features.

In the core (what you would find in ros2.repos), it’s really a limit of maintainer attention and bandwidth. Since packages going out as part of a distribution have a lot of time put into testing, backporting new features (rather than just bugfixes) can potentially be higher risk. ABI/API would be one axis of the risk, which is something this tool could work to address, but we would also need to be mindful of potential interactions with other package functionality/etc.

From a maintainer perspective, by the time the next LTS comes out (eg Lyrical), the previous LTS is two years old and any backport represents a risk of breaking some current behavior/expectation that is built into a downstream users’ workflow. Tools like this would be very helpful in the first year - 18 months of life for a new release, though.

1 Like

I find it a bit sad that there’s no structured way developers could mark packages as ABI-keeping and API-keeping. If there were fields for this in package.xml, it would be much easier to tell which packages follow which strategy.

Aren’t there plans for package format 4? There is already the proposal to extend the license tag with path specifiers. Maybe more has already piled up to trigger thoughts about a new package format?

If you are using the SemVer versioning scheme properly, then it is clear that incrementing the MAJOR version indicates API breakage and that incrementing the MINOR version may indicate ABI changes.

How would you like to communicate API and ABI breakage otherwise? With a dedicated API/ABI level field?

I meant something different. There’s currently no way to explicitly say you follow semver. That’s what I’m asking for. Because currently, this information is either in readme, in quality declaration, or you have to infer it from repo history or changelog.

@christian agree, this comes down to this specification. but most of the application (at least for ROS packages) do not take advantage of the semversions. technically what the application should do is that negotiate the target library version and call the appropriate APIs if possible. so that we can keep the portability of the application… but that is not always manged as designed though.

Leveraging GitHub Actions more sounds great to me.