This topic has been re-hashed several times, so I’ll try to keep this brief (and probably fail to do so).
Two efforts to consider here:
- Supporting standards-based Python packages in ROS tooling (both colcon for local development and buildfarm for released packages)
- Switching existing packages to a standards-based build pipeline
Obviously (2) will require (1). We haven’t even started working on buildfarm support. The colcon-python-project prototype still works as far as I know, and you’re free to give it a try. There are a few sharp edges that you should be aware of:
- Most notably, colcon “symlink installs” don’t entirely work. In particular, any extra “data files” from setuptools are completely missing. The python package itself gets installed, but the data files are used for critical tasks in ROS like ament index registration and launch file installation. If they were getting copied instead of symlinked, we might be able to live with the regressed behavior but their omission is a hard blocker to switching to the new pipeline.
- In some situations, the standards-based build APIs force us to do things that are slow. In certain circumstances, package identification (determining the name, version, and dependencies of a package) is much slower than the current setuptools-only code.
- In addition to slow identification, some packages will take longer to “build” than the setuptools-only code due to the compression/decompression cycle that standards-based APIs force us to perform.
Next steps in colcon’s standards-based development involve folding the functionality from colcon-python-project into colcon-core and hiding it behind a “feature flag” so that it’s in a disabled-by-default state. I’ve already started this process. I don’t think we’ll consider flipping it on by default until we solve the symlinking problem. Eventually the “legacy” setuptools-only Python build pipeline will get yanked out into a separate package so that colcon can continue to function on platforms we support which don’t have new enough Python packages to perform standards-based builds at all.
A fair amount of the friction here is that PyPA is moving at a breakneck pace compared to the operating system distributions. They’re removing features from setuptools that have widespread use on Ubuntu Focal, which we still need to support. It’s hard to be optimistic about the move-fast-and-break-things approach from this perspective.
Now virtual environments are an entirely different challenge. ROS package dependencies installed using rosdep will install Python packages for the system’s default interpreter. When you activate a virtual environment, you’re changing which interpreter is invoked by the python or python3 command. Depending on how you created the virtual environment, this may isolate you from the system packages that rosdep installed. Another complicating factor is that when colcon is installed outside of the virtual environment, the colcon executable will always use the python interpreter it was built with regardless of what virtual environments you’ve activated, and will then use the same interpreter to build packages, again ignoring your virtual environment.
If you simply must use a virtual environment, you can make things work if you go “all-in”. Ignore ALL python packages on your system and install EVERYTHING in a virtual environment, including your build tools and all dependencies. It’s not easy to do.