The security of DigitalOcean customers and their data is a responsibility we approach with utmost dedication. When our customers’ security is threatened we respond swiftly, communicate with transparency, and take accountability, even if the incident root cause occurs beyond the boundaries of DigitalOcean systems.
Today, we wanted to share an update about how DigitalOcean and our customers were impacted by a recent security incident disclosed by Mailchimp. The key takeaways are below:
At 3:30pm ET on August 8th, 2022 transactional emails from our platform, delivered through Mailchimp, stopped reaching our customers’ inboxes. This was discovered by an internal test run by engineering teams to monitor the health of our signup process. We quickly discovered our Mailchimp account had been suspended, with no access, and no other information being provided by Mailchimp. For DigitalOcean, and our customers, this meant email confirmations, password resets, email-based alerts for product health, and dozens of other transactional emails were not reaching their destination.
We received the following communication from Mailchimp:
During that same timeframe on August 8th, our Security Operations team was made aware of a customer who claimed their password had been reset, without their initiation. Recognizing a likely connection between our sudden loss of transactional email, and potentially malicious password resets, which are delivered via email, a security incident and investigation was launched in parallel with the teams addressing our email outage.
One of the first discoveries was a non-DigitalOcean email address that appeared on a regular email from Mailchimp on August 7th. The [@]arxxwalls.com email was not there on a similar Mailchimp email on August 6th. This led us to strongly believe our Mailchimp account was compromised.
Soon after we discovered an issue with our Mailchimp account on August 8th, we initiated contact with Mailchimp, both via traditional support channels and other escalation methods. On August 10th, we had our first actionable response, and conversation with the Mailchimp/Intuit Legal team to better understand the incident and impact on our account. We were formally notified on August 10th by Mailchimp of the unauthorized access to our and other accounts by what we understand to be an attacker who had compromised Mailchimp internal tooling.
The investigation into errant password resets on August 8th led us to a single IP address initiating a password reset against a limited set of DigitalOcean accounts. Our internal logging indicated the attacker IP address x.213.155.164 had successfully changed the password, but in the case below, failed to access the account due to the second-factor authentication on the account. The attacker did not attempt to complete the second factor.
Correlating password reset events from the attacker IP address via our API logging, we confirmed the small number of DigitalOcean accounts targeted by malicious password resets. Though not all resets were successful.
Our security incident response team took action to secure these accounts and have communicated separately with these customers from our broader notification about email address exposure. We can confirm the attacks against DigitalOcean customer account passwords stopped after August 7th.
While the security incident and investigation proceeded, the broader email outage incident management team decided to immediately migrate critical services away from Mailchimp to another email service provider. After working around the clock, our critical transactional emails were back online with another provider at 11pm ET August 9th.
We have three key takeaways from this incident.
Additionally, related but not as a direct result of this incident, we are evaluating two-factor authentication on-by-default for all DigitalOcean customer accounts. If you do not have 2fa enabled on your account, please enable it now.
Each day, our DigitalOcean team logs on with a commitment to acting as responsible data stewards. We understand that our customers’ trust and loyalty comes with transparency and reliability. If, after reading this communication, you have additional questions, please contact us via the information below - a member of our team will be happy to assist.
Thanks, stay safe, and happy coding.