Been spending more time in ROS 2 lately and getting a feel for the daily grind.
For folks working on real robots, what part of going from code change to seeing it run drives you the most nuts? And have you found anything that actually makes it better?
When testing some changes on the robot I’d used to rsync my workspace to the robot, but with a robot running on an ARM platform (jetson for example) that isn’t possible anymore due to different ISA. This means the change to test loop is much longer. I’d like to have a solution for that.
What is also anoying is bug reports that you can’t reproduce. Luckily our customers have gotten quite good at that and provide rosbags with timestamps so finding the issue is much quicker.
For me the most anying part is working on somebody else’s code. It just feels much nicer to work on your own code 
Haha the “working on someone else’s code” one is universal 
The ARM jump sounds rough though. Curious did you try anything to speed that loop back up?
We’ve tried docker multiplatform builds, but that runs qemu under the hood and so is not as quick as you would hope. But it gets the job done.
For CI we’ve added some native ARM runners hosted on Hetzner
without a doubt, compiling things.
afaik because the rclcpp headers end up including roughly 3 bajillion things, you end up with rather long compile times (actually, it might be the linking step, but I haven’t profiled it. it’s one of the two).
add onto that how colcon, cmake, and setuptools/distutils are just annoying to deal with, and the experience is rather sub-par.
I have some rather simple projects which take 30+ seconds to recompile due to one change in one (non-header) file, and that’s with clang & ccache (I tried getting a faster linker like mold to work but could not due to how my ROS environment is set up). with gcc it’d be even longer.
oh, and then of course how colcon slows down the more files & nested subfolders you have because it does a slow re-scanning step every time you run it.
and, on cross-compilation: compilation does not need to be done with the target architecture being the host architecture! clang is able to compile to a given architecture with the host architecture being different perfectly fine with zero issues. it has an option for setting the target triple.
there is zero reason that we need to be dealing with using docker or needing to get a native arm machine just to compile for arm.
or, since ROS controls the entire build process via colcon, why isn’t there something set up for cross compiling, and you just need to install like -aarch64 versions of different packages, then you can build with colcon build --target arm64v8.2a-unknown-linux-gnu for an nvidia jetson orin nano, and then I can just rsync the resulting install directory over to the jetson (perhaps also it may be good to have that when you specify a target, it also sets build-base and install-base to target/[target]/build/ and target/[target]/install/, where target is the previously specified (normalized) target target triple, e.g. target/aarch64v8.2-unknown-linux-gnu/install is the install path for the target triple arm64v8.2-unknown-linux-gnu (arm64 is an alias for aarch64, in llvm).
and because the entire build process is controlled by colcon, it will know all the right arguments to pass to cmake to ensure that it can find all the libraries, headers, etc.
imo, we should be using cross-compilation, not needing to do one of
all of those are dumb solutions. better options exist! why aren’t we using them?!?
Makes sense. Have you actually got the clang cross-compile working end to end or is it more the thing you keep meaning to do? Wondering if the wall is really the deps/sysroot side
For me, it’s the pervasive use of declarative tools. The hard-to-use and debug launch system, popularity of YAML engineering via plugins vs. letting users write their own main, describing behavior via XML instead of code, etc.
Sometimes you just want to use a library to build something new that doesn’t fit in a neat box of existing abstractions. While this is a general software trade-off, I think that the ROS ecosystem could stand to have more diversity of approaches instead of sometimes defending declarative as the only way to do things.
it’s not something I’ve looked too much into, however it’s something that I know it 100% absolutely possible.
you can even cross-compile to macOS by using osxcross (although it is quite annoying that this is required).
I haven’t looked into doing this, as something like this would require a bunch of work from the ROS team in a bunch of different areas. I might make a proposal for something like this at some point if I get time, however.
yeah, I kinda really dislike the whole idea of yaml/toml/xml launch files.
as much as I dislike python, I much prefer the python launch files. even with how awful the api is, I still find it better. however, the api is so annoying that I wrote my own little thing to wrap it based on simple_launch but with significant modifications on top of it.
I have actually explored cross-compilation in ROS2 for ARM architectures quite a bit and even wrote a paper about a redesigned toolchain I created to tackle it: GitHub - TUD-ADS/KRS-Unleashed · GitHub . Toolchain is currently primarily for FPGAs (Kria KR260) but doesn’t matter, the technology for cross-compilation just requires creating a suiteable sysroot. Interestingly ROS already features everything necessary to cross-compile via colcon mixin. Alternatively you can also just use a cmake toolchain file but I prefer the mixin. But I wouldn’t call it perfect as it requires 2 builds (one local to create a mixin file and integrate it and a second to actually cross-compile) and the cross-compile can easily get confused and take a natively build package from the first command for linking
I’ve been thinking about cross compilation more, and I’m really starting to consider making my own tool to do the cross compilation.
it could identify the necessary packages with rosdep, and clang could be used to avoid needing to install a compiler for that specific target (unlike gcc).
I believe that the repo you linked to would require the user to be on ubuntu, which I would like to avoid making a requirement.
honestly, I think the best way to do it would either be via a cmake toolchain. so it could be a mixin which provides a cmake toolchain, providing the cmake toolchain via the cli, or a dedicated tool that provides a cmake toolchain.
you are going to want to use --build-base and --install-base to change the build & install directory that is used for the cross compilation build in order to avoid this. that will ensure they do not get confused, as they don’t share any build files.
There is definitely room for improvement; I actually haven’t so far really touched the original cross-compliation part in a bigger overhaul. I primarly tried to separate and detangle pieces while keeping the original flow working and adding support for further configuration options. I don’t fully understand your issue/approach you try to realize, but yes, I rely on aarch64-linux-gnu-g++ - but this is a simple installable package - my personal experience with rosdep is mixed, as it requires packages to be “well defined” as far as I know - but I would be open to adjust. You can get rid of the “first time” sudo apt install or whatever aarch64-gc.. via clang, but the main issue is that you need to recreate a perfect replica of the file system of your external device system e.g. the sysroot as otherwise dynamic linking,.. will fail - also some packages are not available under ARM and require to be rebuild manually sometimes. Tbh my preferred approach right now is the Embedded Linux/Yocto flow as its clearly defined, doesnt require any patching and bitbake works quite well.
I have not tested another distro, but I am pretty sure once you get ROS 2 and Vitis to work on it, it should also work as everything relies on unix/posix/gnu tools. I dont have support for Windows or Mac as I dont use them and also have no clue how their file system works in detail. Windows cross-compilation I have done so far was either in Eclipse-based IDEs or WSL
the mixin integrates well into the colcon build and allows all sorts of custom arguments - and you can even create different versions like mixin gcc-12 / mixin clang-18 or whatever but i think its primarily taste - toolchain file is also fine, works well with CMake if you want to do some cross-compilation testing outside of the colcon flow as well - but I would argue the indirection via mixin → toolchain is a bit overkill
I do this; but if you run a build command, some packages somehow already end up in the lookup logic - like for example you cross-compile a package that is available native - you already source this via /opt/ros/distro - and if you build low-level packages that extend core-packages,.. this sometimes also results in hickups.