Hi, I’ve been working on building our software stack using only release mode and not building any packages which are test_depend. The problem I’m having is colcon scoops up all the dependencies no matter how they’re marked in the package.xml. I do not use rosdep as I don’t necessarily trust every dev out there chose wisely when building their package.xml anyways, so I’m trying to do this in a more manual way. I don’t believe I should have to build something like ament_cmake_pep257 if I have no plan to build any tests. I also shouldn’t be installing *-dev debian packages in release builds. E.g. a package I have depends on libglib2.0-dev for building, but only needs libglib2.0-0 at runtime, so the process I want is to build the package in release mode, then create a new image with only the release dependencies, and copy over the install/ space to that new image. Colcon though won’t let me separate out those packages that I don’t want to build, even though they are only <test_depend>. Does anyone else do this or have thoughts?
A few thoughts:
Exclude testing and test dependencies
First, I would suggest using -DBUILD_TESTING=OFF to make sure you aren’t pulling in additional dependencies.
Packages typically have testing enabled at the cmake level. In the case that you are attempting to build a package that has additional test dependencies that you have explicitly removed from either your workspace or your base system, you must also explicitly disable building with tests.
Most of the time you will find a cmake stanza like:
if(BUILD_TESTING)
find_package(ament_cmake_pep257 REQUIRED)
# ...
endif()
If you do a “default” cmake build without the dependency installed, this will fail. To explicitly disable testing across the entire workspace, invoke colcon like.
colcon build --cmake-args -DBUILD_TESTING=OFF -DCMAKE_BUILD_TYPE=Release
Or alternatively do it in your cmake builds with
mkdir build && cd build
cmake .. -DBUILD_TESTING=OFF
Filter additional test packages
By default, colcon considers that you want to build everything in the workspace. If you have test packages you would like to exclude, you can either use individual COLCON_IGNORE files (which is kind of cumbersome if you are frequently switching), or use the command-line flags to ignore packages.
colcon build \
--packages-ignore-regex ".*_tests" ".*_mocks" \
--cmake-args -DBUILD_TESTING=OFF -DCMAKE_BUILD_TYPE=Release
Otherwise, your two stage docker build sounds pretty spot on, and a pattern that I have seen. In the first image, install dev dependencies in order to complete to the build, then copy the resulting binaries to a new container with only runtime dependencies installed.
Answering to the part on depending on libfoo instead of libfoo-dev, I think that the best path is probably building a proper Debian package for your own ROS packages:
cd path/to/your/ros/package
bloom-generate rosdebian --os-name ubuntu --ros-distro humble
# then build the Debian package as usual
And then the resulting .deb files, when installed in a target machine via dpkg / apt, will only install the actual binary Debian packages, not the development ones.
It’s the Debian packaging machinery (dh_* programs, etc.) who will take care of finding the actual dependencies, with the help of the ROS-provided templates.
Hope it’s helpful…
I completely get how to handle the cmake portion of the build using the appropriate cmake flags. But the issue still remains that colcon won’t move on to the cmake step until it verifies all packages referenced in the package.xml are there, regardless of what type of dependency they are. I realize the --skip-packages is possible but then I have to re-define once again which dependencies are required. What I really want here is for colcon to completely ignore any dependencies defined as <test_depend> if I’m choosing to build in release mode. I was actually kind of surprised to realize it doesn’t have that capability. Right now if I have, say, ament_cmake_xmllint as a <test_depend>, I cannot build my package in release mode without that package existing.
Yeah I do intend to end up building debians eventually, but why should my release build process which builds those debians have to build any of the test dependencies that won’t get installed in the first place?
Did you try the COLCON_IGNORE approach? Also I think you can use --package-ignore as well instead of skip. It’s a different behavior as anything not discovered by colcon as a package in your workspace will be assumed to be a system dependency and is presumed to be already on your system. It seems that if you ignore any of these packages that you don’t want to install/compile it will get you where you want to go pretty quickly.
colcon historically hasn’t known or cared about the type of a dependency. That has been specifically encoded extra information to facilitate packaging and releasing whereas colcon was focused on being a way to develop in a workspace. And it definitely doesn’t
Most of the packaging and release management tools don’t use colcon or a workspace. For example colcon is never used during the debian build pipeline created by bloom, it’s always standard cmake... make ... make install or setup.py ... if it’s python. And every package is specifically checked out into an isolated container and then with only it’s declared dependencies built in isolation. But for a workspace build we don’t apply this level of rigor.
We could certainly add a filter to exclude specific dependency types where test_depend would be the one that you want to skip. I wouldn’t directly couple it to being in Release mode. As assuming you’ll never run tests in Release mode is not guaranteed. By default in a dev environment testing the Release build is valuable.
Depending on where you’re going this may be something you want to double check that this isn’t turning into an X-Y problem. With the choice to not use rosdep and develop using a different approach you’re already explicitly stepping off of the beaten path.
On this front have you considered defining your own rosdep sources list to refine the keys and keep using the standard tools with your own source of authority.
And if you find any overly broad dependencies or missing dependencies I’m sure the developers would appreciate a PR with the more precise dependencies resolved so we can raise the bar for everyone’s baseline.
Thanks @tfoote I definitely should take a closer look at how bloom performs the debian builds. I would be totally fine with colcon having an option to skip specific dependency types vs it being tied to release builds, I think that solves my problem for sure. I wouldn’t want to COLCON_IGNORE test packages because in some other build pipeline of course I want to build and run tests.
A far as rosdep is concerned I would definitely try to contribute back missing or incorrect dependencies as much as possible. But my end goal here is a completely pinned release build (starting at the base ubuntu docker image) that’s as small as possible. My developers could run a non-pinned “latest” build during development and then when we release we pin everything permanently and have repeatable release builds.