# Security issue on ROS build farm

**URL:** <https://discourse.openrobotics.org/t/security-issue-on-ros-build-farm/9342>\
**Category:** ROS General\
**Created:** [May 30, 2019, 5:45pm UTC](https://discourse.openrobotics.org/t/security-issue-on-ros-build-farm/9342 "2019-05-30T17:45:32Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![gerkey](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/gerkey/32/2322_2.png) [@gerkey](https://discourse.openrobotics.org/u/gerkey)\
**Post date:** [May 30, 2019, 5:45pm UTC](https://discourse.openrobotics.org/t/security-issue-on-ros-build-farm/9342/1 "2019-05-30T17:45:32Z")

</div>

The machine hosting [build.ros.org](http://build.ros.org), which is the build farm for ROS 1, was recently compromised. We took that machine offline and are in the process of deploying a new farm, with the known exploit patched.

The compromise included local privilege escalation sufficient to access the GPG private key used for signing Debian packages and to push Debian packages into the public-facing repository. **We have no reason to believe that any such malicious activity occurred** , but at this point we are unable to rule it out.

So, in an abundance of caution, we are using a newly generated GPG key pair for the new [build.ros.org](http://build.ros.org). Because the same GPG key is used to build ROS 2 packages we also swapped the key on [build.ros2.org](http://build.ros2.org).

As a result of the farm redeployment and key change, users who are installing or updating ROS packages may encounter service disruptions.

We are working diligently to get back to normal operation as quickly as possible and will provide more updates as we have them.

---

<div class="post-metadata">

**Author:** ![kyrofa](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/kyrofa/32/9577_2.png) [@kyrofa](https://discourse.openrobotics.org/u/kyrofa)\
**Post date:** [May 30, 2019, 6:07pm UTC](https://discourse.openrobotics.org/t/security-issue-on-ros-build-farm/9342/2 "2019-05-30T18:07:19Z")

</div>

Thanks for the heads up @gerkey. Given the inability to rule out malicious uploads, can we safely assume the new repo will only consist of newly-rebuilt packages? In which case I expect it’ll be quite a while before everything is back up and running.

---

<div class="post-metadata">

**Author:** ![gerkey](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/gerkey/32/2322_2.png) [@gerkey](https://discourse.openrobotics.org/u/gerkey)\
**Post date:** [May 30, 2019, 6:35pm UTC](https://discourse.openrobotics.org/t/security-issue-on-ros-build-farm/9342/3 "2019-05-30T18:35:35Z")

</div>

@kyrofa, your supposition is correct. We’re in the process of rebuilding the core of ROS 1 Kinetic and will fan out from there once we’re sure that everything is working well. The newly built binaries are going into a new apt repo that will be made available when it’s ready for public consumption.

---

<div class="post-metadata">

**Author:** ![gavanderhoorn](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/gavanderhoorn/32/969_2.png) [@gavanderhoorn](https://discourse.openrobotics.org/u/gavanderhoorn)\
**Post date:** [May 30, 2019, 6:51pm UTC](https://discourse.openrobotics.org/t/security-issue-on-ros-build-farm/9342/4 "2019-05-30T18:51:03Z")

</div>

> [@gerkey](#):
>
> We took that machine offline and are in the process of deploying a new farm, with the known exploit patched.

For those of us running private instances (but perhaps publicly accessible): could you provide a link to a CVE ID or other information about the specific exploit?

---

<div class="post-metadata">

**Author:** ![kyrofa](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/kyrofa/32/9577_2.png) [@kyrofa](https://discourse.openrobotics.org/u/kyrofa)\
**Post date:** [May 30, 2019, 6:55pm UTC](https://discourse.openrobotics.org/t/security-issue-on-ros-build-farm/9342/5 "2019-05-30T18:55:43Z")

</div>

> [@gerkey](#):
>
> The newly built binaries are going into a new apt repo that will be made available when it’s ready for public consumption.

Excellent. The new key is C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654, correct? Let us know if we can help update docs once the repo is ready.

---

<div class="post-metadata">

**Author:** ![gerkey](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/gerkey/32/2322_2.png) [@gerkey](https://discourse.openrobotics.org/u/gerkey)\
**Post date:** [May 30, 2019, 7:01pm UTC](https://discourse.openrobotics.org/t/security-issue-on-ros-build-farm/9342/6 "2019-05-30T19:01:16Z")

</div>

> [@gavanderhoorn](#):
>
> For those of us running private instances (but perhaps publicly accessible): could you provide a link to a CVE ID or other information about the specific exploit?

We believe that the infiltration point was via this vulnerability in the Jenkins Groovy Plugin:

- [NVD - CVE-2019-1003033](https://nvd.nist.gov/vuln/detail/CVE-2019-1003033)

All Jenkins users should update their Groovy Plugin to patch that exploit.

---

<div class="post-metadata">

**Author:** ![gerkey](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/gerkey/32/2322_2.png) [@gerkey](https://discourse.openrobotics.org/u/gerkey)\
**Post date:** [May 30, 2019, 7:05pm UTC](https://discourse.openrobotics.org/t/security-issue-on-ros-build-farm/9342/7 "2019-05-30T19:05:00Z")

</div>

> [@kyrofa](#):
>
> Excellent. The new key is C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654, correct? Let us know if we can help update docs once the repo is ready.

That’s correct, and we’ll definitely take you up on the kind offer for help! But I’d advise against updating anything just yet while things are in flight.

---

<div class="post-metadata">

**Author:** ![nuclearsandwich](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/nuclearsandwich/32/559_2.png) [@nuclearsandwich](https://discourse.openrobotics.org/u/nuclearsandwich)\
**Post date:** [June 4, 2019, 3:59am UTC](https://discourse.openrobotics.org/t/security-issue-on-ros-build-farm/9342/8 "2019-06-04T03:59:14Z")

</div>

We are continuing to investigate the compromise of [build.ros.org](http://build.ros.org), including working with independent third-party security experts. We’ll report back here as we learn more from them. So far we have no indication that the intrusion was anything more than a commodity attack by a group looking to hijack CPU cycles. But we are unlikely to ever be able to completely rule out malicious interference in the ROS binary packaging pipeline. So in an abundance of caution we are (i) continuing to rebuild everything that we reasonably can and (ii) relocating the rest.

## Summary of breaking changes

These changes are coming Thursday or Friday (2019-06-06 or 2019-06-07) this week.

- Signing key used for the primary ROS repositories will change.
- Packages for unsupported rosdistros will move to [snapshots.ros.org](http://snapshots.ros.org).
- The testing repository is being relocated to `ros-testing` from `ros-shadow-fixed`.

### Rebuilding supported distros

We are currently working on full rebuilds of Indigo, Kinetic, Lunar, and Melodic across all architectures on their supported Ubuntu Long Term Support distributions. We expect to have these builds completed by Thursday and available on [packages.ros.org](http://packages.ros.org).

As we fill them in, **[packages.ros.org](http://packages.ros.org) will come to contain only packages built from the bottom up on the newly deployed build farm**. The packages will comprise currently supported ROS distributions (plus Indigo) and only on their respective Ubuntu LTS platform. Debian Stretch packages for Lunar and Melodic will become available some time later.

### Relocating unsupported distros

Packages for the unsupported ROS distributions will be moved to [snapshots.ros.org](http://snapshots.ros.org). Those package indexes will be resigned with a GPG key used specifically for the [snapshots.ros.org](http://snapshots.ros.org) host (this was always a different key from the one used for [packages.ros.org](http://packages.ros.org)). But **we will not be rebuilding binary packages for unsupported ROS distros.** Users should make their own risk assessments regarding whether to use those packages (including the risk of using unsupported software in the first place).

We are currently hoping to make these packages available on [snapshots.ros.org](http://snapshots.ros.org) tomorrow or Wednesday (2019-06-04 or 2019-06-05) so that there is no time when they are completely unavailable but if we have to make a choice between making unsupported packages temporarily unavailable or delaying the deployment of the newly built ROS packages, we’ll choose to deploy the newly built packages.

### Renaming the testing repository

While we’re making these changes to our repository structure we’re also taking the opportunity to rename the testing repository which has been carrying the name “ros-shadow-fixed” to avoid breaking anyone using it. The new name will be “ros-testing” to match the “ros2-testing” repository already available.

* * *

## How to transition

### Adding the new ROS repository key

_Who should do this?_ Everyone who installs ROS packages from [packages.ros.org](http://packages.ros.org)

_When should this be done?_ You can do it right now!

_When **must** this be done?_ Exact time to be determined, but not later than Friday this week (2019-06-07)

_What to do?_  
Set up the new repository key

```bash
sudo apt-key adv --keyserver 'hkp://keyserver.ubuntu.com:80' --recv-key C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654

```

* * *

### Removing the old ROS repository key

_Who should do this?_ Everyone.

_When should this be done?_ After the ROS repositories have been redeployed with the new signing key. Exact time to be determined, but not later than Friday this week (2019-06-07)

_When **must** this be done?_ You should remove it as soon as the ROS repositories are redeployed. Leaving the key trusted won’t cause anything to break but we won’t be using the key to sign anything new in the future so leaving it just leaves you open to possible misuse in the future.

_What to do?_  
Remove the key from your apt keyring

```bash
sudo apt-key del 421C365BD9FF1F717815A3895523BAEEB01FA116

```

* * *

### Migrating to the snapshots repository for unsupported distributions

_Who should do this?_ Anyone who needs to access ROS distributions other than Indigo, Kinetic, Lunar, and Melodic or the Debian Jessie packages for Kinetic

_When should this be done?_ I’ll update everyone here when I’ve created snapshots for all the old ROS releases. After that announcement you can switch to the snapshot repositories.

_When **must** this be done?_ After the ROS repositories have been redeployed with the new signing key. Exact time to be determined, but not later than Friday this week (2019-06-07)

_What to do?_

We’re making some changes to the snapshots repository layouts to support this. I am waiting to add docs until I know they’ll be accurate. In brief the steps will be

1. Add the ROS Snapshots repository key (different from the key above)
2. Add the snapshot repository for your target ROS distribution

* * *

### Updating the testing repository url

_Who should do this?_ Anyone who is currently using the [ros-shadow-fixed](http://wiki.ros.org/ShadowRepository) repository to test ROS packages before a sync.

_When should this be done?_ After the ROS repositories have been redeployed with the new signing key. Exact time to be determined, but not later than Friday this week (2019-06-07)

_When **must** this be done?_ Once the deployment is complete, the shadow-fixed repository will be purged of release contents and you’ll see an error message when updating apt repositories until you update the repository url.

_What to do?_

Check your `/etc/apt/sources.list` and `/etc/apt/sources.list.d/*.list` for `http://packages.ros.org/ros-shadow-fixed` and replace it with `http://packages.ros.org/ros-testing`

---

<div class="post-metadata">

**Author:** ![nuclearsandwich](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/nuclearsandwich/32/559_2.png) [@nuclearsandwich](https://discourse.openrobotics.org/u/nuclearsandwich)\
**Post date:** [June 4, 2019, 3:59am UTC](https://discourse.openrobotics.org/t/security-issue-on-ros-build-farm/9342/9 "2019-06-04T03:59:29Z")

</div>

> [@kyrofa](#):
>
> Let us know if we can help update docs once the repo is ready.

This is definitely help we’ll need going into the weekend. I’ve opened [Updates for ROS repository changes 2019-06 · Issue #276 · ros-infrastructure/roswiki · GitHub](https://github.com/ros-infrastructure/roswiki/issues/276) to track documentation that will need updating after the changes above.

---

<div class="post-metadata">

**Author:** ![gavanderhoorn](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/gavanderhoorn/32/969_2.png) [@gavanderhoorn](https://discourse.openrobotics.org/u/gavanderhoorn)\
**Post date:** [June 4, 2019, 7:09am UTC](https://discourse.openrobotics.org/t/security-issue-on-ros-build-farm/9342/10 "2019-06-04T07:09:04Z")

</div>

Just adding this as I don’t see it mentioned anywhere: anyone who is building custom Docker images (or using any other sort of containerisation technology) and does _not_ start from any of the official `ros` or `osrf/ros` images will also have to update their `Dockerfile`s (or whatever it is called for the tool you are using) to use / import the new keys.

---

<div class="post-metadata">

**Author:** ![Martin\_Guenther](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/martin_guenther/32/91_2.png) [@Martin\_Guenther](https://discourse.openrobotics.org/u/Martin_Guenther)\
**Post date:** [June 5, 2019, 8:56am UTC](https://discourse.openrobotics.org/t/security-issue-on-ros-build-farm/9342/11 "2019-06-05T08:56:37Z")

</div>

Also don’t forget to update your `.travis.yml` / `.gitlab-ci.yml` etc. files if they contain the string `421C365` or `ros-shadow-fixed`.

---

<div class="post-metadata">

**Author:** ![nuclearsandwich](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/nuclearsandwich/32/559_2.png) [@nuclearsandwich](https://discourse.openrobotics.org/u/nuclearsandwich)\
**Post date:** [June 6, 2019, 12:36am UTC](https://discourse.openrobotics.org/t/security-issue-on-ros-build-farm/9342/12 "2019-06-06T00:36:31Z")

</div>

The Snapshots repository now has `final` snapshots of all unsupported ROS distributions and distributions with unsupported platforms (i.e. Kinetic on Jessie) with the exception of Indigo on Trusty which is being rebuilt and will therefore be on [packages.ros.org](http://packages.ros.org) for a while longer as well as having a final snapshot made when it’s done.

If you’re using one of these unsupported rosdistros. I’d recommend updating your repositories now so you can let us know if you run into any difficulties. We are still on track to sync the updated repositories this Friday (2019-06-07).

Information on using the snapshot repository can be found on the ROS wiki [SnapshotRepository - ROS Wiki](http://wiki.ros.org/SnapshotRepository)

---

<div class="post-metadata">

**Author:** ![B0gdar](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/b0gdar/32/6119_2.png) [@B0gdar](https://discourse.openrobotics.org/u/B0gdar)\
**Post date:** [June 6, 2019, 7:51pm UTC](https://discourse.openrobotics.org/t/security-issue-on-ros-build-farm/9342/13 "2019-06-06T19:51:01Z")

</div>

Wouldn’t this be a great candidate to feature on the [ROS news section](http://www.ros.org/news/) also? There may be a great set of users from old times that may not be regularly checking the discourse threads…

Thank you for your hard work guys!

---

<div class="post-metadata">

**Author:** ![tfoote](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/tfoote/32/3813_2.png) [@tfoote](https://discourse.openrobotics.org/u/tfoote)\
**Post date:** [June 7, 2019, 7:46pm UTC](https://discourse.openrobotics.org/t/security-issue-on-ros-build-farm/9342/14 "2019-06-07T19:46:56Z")

</div>

We have rebuilt all debian packages for indigo, kinetic, lunar, and melodic. And the packages have been redeployed with the updated GPG key to [packages.ros.org](http://packages.ros.org) see the announcement: [New GPG keys deployed for packages.ros.org - ROS General - Open Robotics Discourse](http://discourse.ros.org/t/new-gpg-keys-deployed-for-packages-ros-org/9454)

---

<div class="post-metadata">

**Author:** ![gerkey](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/gerkey/32/2322_2.png) [@gerkey](https://discourse.openrobotics.org/u/gerkey)\
**Post date:** [June 7, 2019, 8:50pm UTC](https://discourse.openrobotics.org/t/security-issue-on-ros-build-farm/9342/15 "2019-06-07T20:50:32Z")

</div>

> [@tfoote](#):
>
> We have rebuilt all debian packages for indigo, kinetic, lunar, and melodic.

That is great news.

I’d like to take this chance to extend a **huge thank you** to my colleagues who moved heaven and earth over the past week and a half to respond to this security compromise and get our systems back into a healthy state for the entire community. In particular, @tfoote, @nuclearsandwich, and @jrivero put in many many hours over nights and through the weekend, so cheers to them!

Also a big thanks to @Thomas_Moulard and his colleagues at AWS who helped along the way with diagnosis and mitigation of the initial problem and planning for the redeployment, and also served as our early testers of the outputs from the new farm.

And thank you to everyone in the community for your patience and support as we’ve worked through an unfortunate incident.

---

<div class="post-metadata">

**Author:** ![bionade24](https://avatars.discourse-cdn.com/v4/letter/b/8491ac/32.png) [@bionade24](https://discourse.openrobotics.org/u/bionade24)\
**Post date:** [June 30, 2019, 5:21pm UTC](https://discourse.openrobotics.org/t/security-issue-on-ros-build-farm/9342/16 "2019-06-30T17:21:56Z")

</div>

Here a jenkins security advice: [securing jenkins](http://turnoff.us/geek/securing-jenkins/)