# SSL failure in buildfarm jobs (ROS1)
**URL:** https://discourse.openrobotics.org/t/ssl-failure-in-buildfarm-jobs-ros1/18304
**Category:** Buildfarm
**Created:** [January 7, 2021, 9:21am UTC](https://discourse.openrobotics.org/t/ssl-failure-in-buildfarm-jobs-ros1/18304 "2021-01-07T09:21:56Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@xqms](https://discourse.openrobotics.org/u/xqms)
#### Post date: [January 7, 2021, 9:21am UTC](https://discourse.openrobotics.org/t/ssl-failure-in-buildfarm-jobs-ros1/18304/1 "2021-01-07T09:21:56Z")
Hi, I’m seeing strange, consistent failures of the armhf job for my rosfmt package since a few days:
[https://build.ros.org/job/Nbin\_ufhf\_uFhf\_\_rosfmt\_\_ubuntu\_focal\_armhf\_\_binary/11/console](https://build.ros.org/job/Nbin_ufhf_uFhf __rosfmt__ ubuntu_focal_armhf__binary/11/console)
It seems it’s failing to download the rosdistro index ([https://raw.githubusercontent.com/ros/rosdistro/master/index-v4.yaml](https://raw.githubusercontent.com/ros/rosdistro/master/index-v4.yaml)) because it cannot verify the SSL certificate. I think it’s because the Docker container does not have the appropriate root certificates installed, but I’m not sure. Does anyone have a similar issue or knows what’s going on?
The strange thing is that it affects only the armhf build…
---
### Author:  [@gavanderhoorn](https://discourse.openrobotics.org/u/gavanderhoorn)
#### Post date: [January 7, 2021, 9:53am UTC](https://discourse.openrobotics.org/t/ssl-failure-in-buildfarm-jobs-ros1/18304/2 "2021-01-07T09:53:49Z")
Perhaps related: [osrf/multiarch-docker-image-generation#38](https://github.com/osrf/multiarch-docker-image-generation/issues/38)?
* * *
Edit: probably not: that particular `armhf` build appears to run on an `amd64` agent.
---
### Author:  [@mikeferguson](https://discourse.openrobotics.org/u/mikeferguson)
#### Post date: [January 8, 2021, 4:22am UTC](https://discourse.openrobotics.org/t/ssl-failure-in-buildfarm-jobs-ros1/18304/3 "2021-01-08T04:22:23Z")
Also seeing the same failure on a few recent releases:
- [https://build.ros.org/job/Nbin\_ufhf\_uFhf\_\_arbotix\_firmware\_\_ubuntu\_focal\_armhf\_\_binary/4/console](https://build.ros.org/job/Nbin_ufhf_uFhf __arbotix_firmware__ ubuntu_focal_armhf__binary/4/console)
- [https://build.ros.org/job/Nbin\_ufhf\_uFhf\_\_arbotix\_msgs\_\_ubuntu\_focal\_armhf\_\_binary/4/console](https://build.ros.org/job/Nbin_ufhf_uFhf __arbotix_msgs__ ubuntu_focal_armhf__binary/4/console)
---
### Author:  [@nuclearsandwich](https://discourse.openrobotics.org/u/nuclearsandwich)
#### Post date: [January 13, 2021, 4:55am UTC](https://discourse.openrobotics.org/t/ssl-failure-in-buildfarm-jobs-ros1/18304/4 "2021-01-13T04:55:35Z")
@cottsay did an investigation of this issue last week and identified that for as yet undetermined reasons, `openssl rehash` is not doing its job in our containers but that running `c_rehash` is enough to kick things into shape.
His minimal reproduction, which I’ve been using to test, involves installing openssl and ca-certificates and running this test command:
```auto
echo "" | openssl s_client -connect raw.githubusercontent.com:443 | grep "Verification: OK" || echo "FAILED"
```
QEMU makes tracing the rehash process in the “guest” container impossible and I wasn’t able to reproduce the problem using our image or the stock `ubuntu:focal` image on native hardware. I was also able to produce the issue using the “stock” cross platform image with qemu injected.
```auto
docker run -ti -v /usr/bin/qemu-arm-static:/usr/bin/qemu-arm-static --platform armhf ubuntu:focal bash
```
As a temporary workaround we added [this patch](https://github.com/ros-infrastructure/ros_buildfarm/pull/848) which runs `c_rehash` in the armhf containers but it now looks like we’re getting new errors that weren’t there before when actually building. I’ll be digging into those tomorrow with the goal being to get us back to building and then circle back to figuring out what exactly is happening.
---
### Author:  [@nuclearsandwich](https://discourse.openrobotics.org/u/nuclearsandwich)
#### Post date: [January 13, 2021, 5:13am UTC](https://discourse.openrobotics.org/t/ssl-failure-in-buildfarm-jobs-ros1/18304/5 "2021-01-13T05:13:26Z")
There aren’t Debian armhf builds on the farm but I can see this problem with Debian buster and bullseye containers as well.
---
### Author:  [@nuclearsandwich](https://discourse.openrobotics.org/u/nuclearsandwich)
#### Post date: [January 14, 2021, 6:20am UTC](https://discourse.openrobotics.org/t/ssl-failure-in-buildfarm-jobs-ros1/18304/6 "2021-01-14T06:20:57Z")
For a summary of day two, we haven’t identified the exact problem. When running `/usr/bin/qemu-arm-static -strace /usr/bin/openssl rehash` to see the interpreted system calls the problem does not exhibit! Making this a very textbook [heisenbug](https://en.wikipedia.org/wiki/Heisenbug). Interestingly, invoking the interpreter explicitly without `-strace` (`/usr/bin/qemu-arm-static /usr/bin/openssl rehash`) also doesn’t exhibit the issue.
The issue doesn’t present on an 18.04 host. I didn’t have 18.10, 19.04, or 19.10 images to test with but it does seem like an interaction between the host kernel/libc, qemu, and the container userspace.
Once we we’d gotten that far we switched to trying a second workaround. With the 20.04 deployment one of the things we enabled were native ARM agents. When testing them we found an issue with the bundled default seccomp policy for Ubuntu 20.04, which also reminded me to include the personality system call used by sbcl. [That change](https://github.com/ros-infrastructure/cookbook-ros-buildfarm/pull/39) has been added to our native ARM machine and will soon be part of the cookbook. With it we were able to get a successful build of [https://build.ros.org/job/Nbin\_ufhf\_uFhf\_\_rosfmt\_\_ubuntu\_focal\_armhf\_\_binary/21/](https://build.ros.org/job/Nbin_ufhf_uFhf __rosfmt__ ubuntu_focal_armhf__binary/21/)
and when [Fix label configuration for noetic armhf. by nuclearsandwich · Pull Request #195 · ros-infrastructure/ros\_buildfarm\_config · GitHub](https://github.com/ros-infrastructure/ros_buildfarm_config/pull/195) is deployed we’ll be using the native ARM agents for all our Noetic armhf builds.
I’m going to write up what we’ve got on the qemu-discuss list tomorrow hoping for advice or suggesting we’ve found a bug to formally report. If the trail goes cold there we may not have availability to continue investigating the issue when running within QEMU as long as native agents are working for us.