A Message About Intel Security Findings

Update Tuesday, April 17th, 2018:

Today we’re excited to share that we have completed the reboot process in our NYC2 datacenter, wrapping up our Spectre and Meltdown mitigation efforts. Rebooting activity across our fleet of 12 datacenters is now complete!

If you experience any issues with a Droplet that was rebooted during these mitigation efforts, please refer to this Community article we posted to help you troubleshoot.

We appreciate your patience and understanding throughout the duration of this necessary maintenance.

Update Thursday, April 12, 2018:

We’re happy to share that today we have successfully completed reboot activity for all customer hypervisors in our AMS2 datacenter.

We anticipate completing the reboot process in our NYC2 datacenter early next week. Users can expect email notifications about the maintenance window for their impacted Droplets at least 24 hours ahead of the scheduled reboots.

The completion of reboots in our NYC2 datacenter will also mark the completion of reboots for our entire fleet, and we will share an update here as soon as this on-going maintenance is finished.

Update Tuesday, April 3, 2018:

We’re happy to share that today we have successfully completed reboot activity for all customer hypervisors in our SFO1 datacenter.

Next week, we are continuing reboots, with maintenance in our AMS2 and NYC2 regions. The reboot process for our entire fleet will continue over the coming weeks. Users can expect email notifications about the maintenance window for their impacted Droplets at least 24 hours ahead of the scheduled reboots.

We will continue to share progress updates here and alert our users to the completion of reboots in each of our regions as information becomes available.

Update Wednesday, March 28, 2018:

We’re happy to share that today we have successfully completed reboot activity for all customer hypervisors in our SGP1 and FRA1 datacenters.

We are continuing reboots in our SFO1 region, and we expect that maintenance to be completed early next week. The reboot process for our entire fleet will continue over the coming weeks. Users can expect email notifications about the maintenance window for their impacted Droplets at least 24 hours ahead of the scheduled reboots.

We will continue to share progress updates here and alert our users to the completion of reboots in each of our regions as information becomes available.

Update Friday, March 23, 2018:

We’re happy to share that today we have successfully completed reboot activity for all customer hypervisors in our LON1 datacenter.

Reboots in SGP1 are well underway, and next week we will also conduct them in the FRA1 and SFO1 datacenters. Users can expect email notifications about the maintenance window for their impacted Droplets at least 24 hours ahead of the scheduled reboots.

The reboot process for our entire fleet will continue over the coming weeks. We will share progress updates here and alert our users to the completion of reboots in each of our regions as information becomes available.

Update Wednesday, March 21, 2018:

We’re happy to share that today we have successfully completed reboot activity for all customer hypervisors in our NYC3 datacenter.

Reboots in SGP1 and LON1 are currently underway and we continue to coordinate reboots for our other datacenters. The reboot process for our entire fleet will continue over the coming weeks. Users can expect email notifications about the maintenance window for their impacted Droplets at least 24 hours ahead of the scheduled reboots.

We will continue to share progress updates here and alert our users to the completion of reboots in each of our regions as information becomes available.

Update Monday, March 12, 2018:

We’re happy to share that today, we have successfully completed reboot activity for all customer hypervisors in our AMS3 datacenter.

We continue to coordinate reboots for our other datacenters, with SGP1 planned for the next maintenance window, and the reboot process for our entire fleet will continue over the coming weeks. Users can expect email notifications about the maintenance window for their impacted Droplets at least 24 hours ahead of the scheduled reboots.

We will continue to share progress updates here to alert our users to the completion of reboots in each of our regions, or if new information becomes available as we work through our rebooting schedule.

Update Thursday, March 1, 2018:

Last week, we began rebooting Droplets in our SFO2 and BLR1 datacenters. We’re happy to share that today we successfully completed reboot activity for all customer hypervisors in the SFO2 and BLR1 datacenters. This week, we also started and finished rebooting activity in TOR1, and we are continuing to reboot Droplets in our NYC3 datacenter.

The reboot process for our entire fleet will continue over the coming weeks. Users can expect email notifications about the maintenance window for their impacted Droplets at least 24 hours ahead of the scheduled reboots.

We will continue to share progress updates here to alert our users to the completion of reboots in each of our regions, or if new information becomes available as we work through our rebooting schedule.

Update Friday, February 16, 2018:

Last week, we began rebooting Droplets in our NYC1 datacenter. We’re happy to share that we have successfully completed reboot activity for all customer hypervisors in this datacenter.

Reboots in our NYC3 datacenter are underway, and next week we will also begin reboot maintenance in the BLR1 and SFO2 datacenters. We anticipate activity lasting for two days in BLR1 (Tuesday 2/20 and Wednesday 2/21) and three business days in SFO2 (Wednesday 2/21 through Monday, 02/26). We will continue the reboot process for our entire fleet over the coming weeks. Users can expect email notifications about the maintenance window for their impacted Droplets at least 24 hours ahead of the scheduled rebooting.

Moving forward, we will share progress updates here to alert our users to the completion of reboots in each of our regions, or if new information becomes available as we work through our rebooting schedule.

Update Friday, February 9, 2018:

This week we began rebooting Droplets in our NYC1 datacenter. The maintenance is going well and we will continue the reboot process for our entire fleet over the coming weeks. Users can expect email notifications about the maintenance window for their impacted Droplets at least 24 hours ahead of the scheduled rebooting.

While these rebooting efforts are necessary to apply the patches that mitigate the Spectre and Meltdown vulnerabilities within DigitalOcean’s infrastructure, users should also apply patches in their Droplets to achieve complete protection. To ensure your Droplets are as secure as possible, we recommend that you follow this tutorial to protect against the Meltdown and Spectre vulnerabilities. We also recommend taking a backup or snapshot of critical data before making changes to a production system.

Mitigating the risks presented by Spectre and Meltdown is a top priority for our engineering team and we are working hard to minimize disruption during this necessary maintenance. During this process we will communicate with you in the following ways:

• Status page updates related to the scheduled maintenance and separate status reports if issues arise
• Email notifications, including a list of affected Droplets, to all affected customers at least 24 hours ahead of scheduled maintenance windows
• Blog updates as new information becomes available

Update Friday, February 2, 2018:

This week, our engineering and infrastructure teams completed the preparation and testing necessary to begin our planned reboots on Monday, February 5th.

The reboots will be done on a rolling basis and will affect all Droplets in all regions. We will be starting reboots in the NYC1 region and have notified all customers who will be affected on Monday and Tuesday by email. We will continue to notify affected customers at least 24 hours in advance as we reboot their physical machines and the Droplets on them. These reboots are necessary in order to apply the patches that mitigate the Spectre and Meltdown vulnerabilities within DigitalOcean’s infrastructure.

In some cases, patching inside Droplets may be more critical than others. We encourage users to determine the best course of action and strongly recommend you follow the steps outlined in this article to improve your security and ensure your Droplet is running an updated kernel. We also recommend taking a backup or snapshot of critical data before making changes to a production system.

If your distribution is not included in the list of patched versions below, we highly recommend you move your data to a new Droplet running a version that is receiving security updates. To simplify the act of patching, we have recently updated Droplets to utilize a GrubLoader, which allows Droplets that use our in-control panel legacy system to boot into internally installed kernels. On certain legacy Droplets, this may cause issues if the kernel is not upgraded.

Meltdown distribution updates to date: CentOS 6 & 7, RancherOS, CoreOS, Debian 7, 8 & 9, Fedora 27 Atomic, Ubuntu 14.04, 16.04 & 17.10 base images have been updated. Fedora 26 & 27 have updates available, but users will have to manually update as Fedora does not have updated Cloud Images. Our Ubuntu 16.04 1-Click images have been refreshed to include the latest patches and updates.

Spectre distribution updates to date: At this time only Ubuntu and CentOS have released kernel updates to address Spectre Variant 1 and they have been included in our CentOS 6 & 7 base images. Both kernels include Spectre Variant 2 fixes, however, they are not enabled on our Cloud Platform at this time.

We will work as hard as possible to minimize disruption during these reboots. During this process we will communicate with you in the following ways:

• Status page updates related to the scheduled maintenance and separate status reports if issues arise
• Email notifications, including a list of affected Droplets, to all affected customers at least 24 hours ahead of scheduled maintenance windows
• Blog updates as new information becomes available

Update Friday, January 26, 2018:

This week our engineering team completed testing of our candidate kernel with all of our major subsystems and tested the fixes that will be deployed across our fleet.

We are ready to begin our planned reboots which will affect all Droplets in all regions, and have notified affected customers by email. During the course of this maintenance, we will reboot physical machines and the Droplets on them.

In some cases patching inside Droplets may be more critical than others. We encourage users to determine the best course of action and we strongly recommend you follow the steps outlined in this article to improve your security and ensure your Droplet is running an updated kernel. We also recommend taking a backup or snapshot of critical data before making changes to a production system.

If your distribution is not included in the list of patched versions listed in our January 19th update, we highly recommend you move your data to a new Droplet running a version that is receiving security updates. To simplify the act of patching, we have recently updated Droplets to utilize a GrubLoader. On certain legacy Droplets, this may cause issues if the kernel is not upgraded.

We will be starting reboots in NYC1 as early as Wednesday, January 31st and we will be communicating with customers in the following ways:

• Email notifications, including a list of affected Droplets, will be sent to all affected customers at least 24 hours ahead of scheduled maintenance windows.
• We will be using our status page to communicate about any reboot-related incidents.
• We will continue to share updates on this blog as we have them.

Update Friday, January 19, 2018:

Over the past week, our engineering team has identified and begun formal testing on a set of kernel patches that begin to mitigate all three variants. We are validating this candidate kernel with all of our major subsystems and starting to plan and test this initial round of fixes to deploy across our fleet.

We anticipate a robust testing phase over the next week to ensure these changes will not negatively impact our customers, continuing our approach of taking careful, well-informed steps towards long term resolution, rather than a string of one-off mitigation efforts.

We plan to provide our next update on Friday, January 26th. If we determine that we are able to initiate reboots sooner, we will provide an update here and e-mail affected customers directly with at least 24 hours advance notice.

In the meantime, we encourage you to ensure your servers are as secure as possible. For more information about protecting your Droplets, you can reference this tutorial.

Meltdown distribution updates to date: CentOS 6 & 7, RancherOS, CoreOS, Debian 7, 8 & 9, Fedora 27 Atomic, Ubuntu 14.04, 16.04 & 17.10 base images have been updated. Fedora 26 & 27 have updates available, but users will have to manually update as Fedora does not have updated Cloud Images. Our Ubuntu 16.04 1-Click images have been refreshed to include the latest patches and updates.

Spectre distribution updates to date: At this time only CentOS 6 and CentOS 7 have released kernel updates to address Spectre Variant 1 and have been included in our our CentOS 6 & 7 base images.

Update Friday, January 12, 2018:

Our engineering team continues to procure and test patches as they become available and we have a significant amount of resources dedicated to this task. While numerous patches have been rolled out for Meltdown, mitigations for Spectre are still sparse and raw. Unfortunately, many distributions have not been able to roll out a full set of patches to address all 3 variants of the exploits. We will continue preliminary testing through the week of January 15th. These tests will have no customer impact, and will be focused on getting machines up and running in these new environments.

Intel released a microcode update this week, unfortunately the update was determined to cause stability issues for other Intel customers, and has since been pulled back. DigitalOcean did not apply this microcode to our fleet, and we are awaiting the release of new a microcode. Once we have the final microcode in hand, we will begin performance/regression testing to validate the update in our environment.

As we have previously mentioned, fleet wide reboots will take place following successful testing and validation. We will communicate the reboot schedule to customers in advance of any action. In the meantime, we expect to share another update here on Friday 1/19. For more information about protecting your Droplets, you can reference this tutorial.

Meltdown distribution updates to date: CentOS 6 & 7, RancherOS, CoreOS, Debian 7, 8 & 9, Fedora 27 Atomic, Ubuntu 14.04, 16.04 & 17.10 base images have been updated. Fedora 26 & 27 have updates available, but users will have to manually update as Fedora does not have updated Cloud Images.

Update Tuesday, January 9, 2018:

As the ongoing security vulnerability developments evolve there are still many unknowns. Like many other cloud service providers we are participating in Linux kernel working groups, coordinating with Intel and other hardware vendors, and doing our own exhaustive research. The goal is to protect the security of our users’ data and provide a long term solution instead of offering a cascade of short term fixes. That said, here is our approach based on what we know today.

As mitigations for vulnerabilities are released, our engineering team is diligently and methodically testing each one to ensure that our customers have stability and performance when the patches are applied. We will continue this testing process for all new patches. It is difficult to estimate the timeframe we’ll need to create, debug and test them, as new patches are being rolled out sporadically, but we anticipate that the testing phase will last for at least another week. We plan to share another update this Friday, January 12th. As mentioned in our last post, we will alert customers in advance of any reboots that need to take place.

In the meantime, we encourage you to track the patches being released on your distributions and we’ve compiled a list of distribution patches released thus far, which we will update as they become available. It’s important to note that updated distributions do have various fixes, but none have remediations for all three vulnerabilities. In order to help our users protect themselves as the patches become available, we have changed all Droplets to utilize our Grubloader kernel, ensuring that Droplet kernels can be upgraded by the user, without DigitalOcean involvement.

Meltdown distribution updates to date:
CentOS 7, RancherOS, CoreOS, Debian 9, Fedora 27 Atomic

We’ve also authored this tutorial to help you apply patches. This, too, will be updated as more information and patch releases become available.

Update Friday, January 5, 2018:

Our engineering team continues to remain in close coordination with Intel, Canonical, and our other vendors. We are currently awaiting patches that, once applied, should mitigate the security vulnerabilities. We expect to have those patches on Tuesday, January 9th, and will begin formal testing as soon as they are received.

In the interim, as patches become available on the Linux kernel list and microcode updates become available from Intel and other vendors, we are doing ad-hoc testing to understand potential performance implications and evaluate stability concerns so we can execute our mitigation as smoothly as possible.

The scope of work is extensive; everything from the kernel to compilers and emulation systems must be patched and tested. We will be devoting all of our available engineering resources to this effort, but the set of changes is so significant that we cannot yet estimate the time frame needed to complete validation. The security of our customers and the reliability of our services are important to us and it is critical that we take the time to validate before we roll out changes.

We expect to post another update on Tuesday, January 9th, once we have received the patches and testing is underway. We will share updates here sooner if additional information becomes available. We appreciate your patience!

Update Thursday, January 4, 2018:

Our engineering team continues to coordinate closely with Intel to determine the exact scope and impact of the Meltdown and Spectre security vulnerabilities. It is our current understanding that DigitalOcean is not vulnerable to the Meltdown (Variant 3) exploit because of our usage of KVM virtualization. However, we will still be taking the necessary steps to protect our customers from the impact of the Spectre (Variants 1 and 2) exploits.

We will be obtaining the patches necessary to mitigate the vulnerabilities and once our engineering team has validated them, we will be rebooting our entire fleet of Droplets. DigitalOcean users will also need to upgrade their own kernels, and we will be working closely with them to ensure that this process goes as smoothly as possible. Every customer will receive advanced notification before we initiate the reboots.

Original post Wednesday, January 3, 2018:

Earlier this week, we became aware of a potential security flaw impacting Intel hardware used by DigitalOcean and many other cloud service providers. Since learning of this issue, we have been actively investigating and tracking Linux kernel activity and our development team has been working diligently to obtain as much information as possible from Intel. Unfortunately, the strict embargo placed by Intel has significantly limited our ability to establish a comprehensive understanding of the potential impact.

Based on our investigation and the information we have received thus far, we believe that it may be necessary to reboot impacted customer Droplets. If reboots are determined to be the correct course of action for DigitalOcean users, we will schedule the urgent maintenance and notify impacted customers in advance.

We are continuing to monitor this situation and work with Intel to obtain more details. We’ll share updates in this blog post as additional information becomes available to us.

You can read Intel’s initial statement here.

Josh Feinblum leads security and compliance for DigitalOcean and serves as Chief Security Officer. Prior to DigitalOcean, he was the head of security at Rapid7 and started several security programs across hyper-growth, technology-oriented healthcare companies. He is deeply involved in the security community and has more than 14 years of experience managing security teams, overseeing major clients at large managed service providers, and starting privacy and security related programs across commercial and federal financial service firms.