Over the last few weeks, some customers have reported intermittant performance issues with Spaces in AMS3. Our engineering team identified two primary causes: the way the cluster manages available system memory and the overall load on the cluster.
We're taking the following actions:
On 4 June, we deployed a fix that resolves the memory management issues.
Starting at 9pm UTC on 19 July, we will be performing maintenance on the cluster to alleviate the load issues. We will be temporarily disabling the creation of new Spaces in AMS3 until this maintenance is complete.
During the maintenance, existing Spaces in AMS3 will continue to serve content as normal, and you can still create new Spaces in any other Spaces-compatible datacenter.
Starting at 9pm UTC on 19 July, we may also selectively throttle some operations from Spaces in AMS3 for a limited number of customer accounts.
This will not affect most accounts, and we will reach out beforehand if we identify your Space as a candidate.
In November of this year, we plan to expand AMS3's capacity as a long-term solution to handle the cluster's load.
We will continue to update this page with our progress on the AMS3 maintenance and expansion.
Starting in September, we are applying an allowance and bandwidth rate limit on uploads to existing Spaces in AMS3:
Uploading up to 150 GB of data in 24 hours is unthrottled.
If uploads to a Space exceed 150 GB of data in 24 hours, that Space's upload bandwidth is gradually throttled until its 24-hour accumulated uploads return to 150 GB or less.
If uploads to a Space exceed 270 GB of data in 24 hours, that Space's upload bandwidth is set to 1MB/s for the next 24 hours.
The upload allowance and rate limiting will remain in place until the datacenter expansion later this year.
Due to unforeseen delays, we were not able complete AMS3's capacity expansion by November as originally planned.
Our revised timeline for completing the region's new cluster deployment is February 2020, at which point we'll lift the temporary limits on upload allowance and bandwidth rates and reenable the creation of new Spaces in AMS3.