For a summary of day two, we haven’t identified the exact problem. When running /usr/bin/qemu-arm-static -strace /usr/bin/openssl rehash to see the interpreted system calls the problem does not exhibit! Making this a very textbook heisenbug. Interestingly, invoking the interpreter explicitly without -strace (/usr/bin/qemu-arm-static /usr/bin/openssl rehash) also doesn’t exhibit the issue.
The issue doesn’t present on an 18.04 host. I didn’t have 18.10, 19.04, or 19.10 images to test with but it does seem like an interaction between the host kernel/libc, qemu, and the container userspace.
Once we we’d gotten that far we switched to trying a second workaround. With the 20.04 deployment one of the things we enabled were native ARM agents. When testing them we found an issue with the bundled default seccomp policy for Ubuntu 20.04, which also reminded me to include the personality system call used by sbcl. That change has been added to our native ARM machine and will soon be part of the cookbook. With it we were able to get a successful build of https://build.ros.org/job/Nbin_ufhf_uFhf__rosfmt__ubuntu_focal_armhf__binary/21/
and when Fix label configuration for noetic armhf. by nuclearsandwich · Pull Request #195 · ros-infrastructure/ros_buildfarm_config · GitHub is deployed we’ll be using the native ARM agents for all our Noetic armhf builds.
I’m going to write up what we’ve got on the qemu-discuss list tomorrow hoping for advice or suggesting we’ve found a bug to formally report. If the trail goes cold there we may not have availability to continue investigating the issue when running within QEMU as long as native agents are working for us.