Performance Tips for DigitalOcean Spaces

Here are some recommendations on how to get the best performance from Spaces based on your use case and application architecture.

Use a Content Delivery Network (CDN)

When Should I Do This?

A CDN, or content delivery network, caches your assets in geographically distributed locations to make downloads and page loads faster for your users.

You should use a CDN in front of Spaces if:

  • Your use case is mostly serving GET requests from the Internet, especially for frequent requests to small files.

For example, web-facing applications and media servers are especially likely to see significant performance improvements with the addition of a CDN.

How Does This Improve Performance?

By integrating a CDN with Spaces, you can distribute content to your users with low latency and a higher data transfer rate than you could get by serving your content directly from Spaces.

A CDN will fetch requested files from Spaces and cache them closer to your end users. By serving future requests for the same files from the CDN’s cache, you reduce the number of GET requests sent to Spaces and therefore reduce the user’s overall latency when interacting with your application.

How Do I Implement This?

You can use the free Spaces CDN, which is available as part of your Spaces subscription at no additional cost.

Several third party CDNs have documentation specifically for Spaces, like Fastly and KeyCDN, and most other CDNs will work with Spaces with minimal configuration. However, in order to use Cloudflare as a CDN, you need to either use a Cloudflare worker (which is a paid service) or use a tier that supports host header rewrites (which the free tier currently does not).

Avoid Small Files and Use Multi-Part Uploads for Large Files

When Should I Do This?

You should consider the size of your files and the way you upload them to Spaces if you are:

  • Handling files smaller than 1MB.
  • Uploading files larger than 500MB.

How Does This Improve Performance?

Spaces is designed for storing and serving moderate to large files. Files 20MB - 200MB in size will give the best performance for both writes and reads. Additionally, combining small files into one larger file will greatly reduce the overall number of requests to your Space compared to handling many small files individually.

How Do I Implement This?

When uploading files larger than 500MB, you should use multi-part uploads.

We recommend combining files less than 1MB together into a single, larger file. How you do this will be specific to your particular files and use case, but one example is concatenating daily log files into a monthly file.

Choose the Right Data Center for Your Resources

When Should I Do This?

Choosing the right datacenter location for your Spaces depends on where the connections to your Spaces come from.

If the connections to your Spaces are from Droplets, you’ll see the best performance when you:

  • Ideally, put your Droplets and Spaces in the same datacenter.
  • Alternatively, put your Droplets and Spaces in datacenters connected by DigitalOcean’s global backbone.

If the connections to your Spaces are from end users on the Internet, you’ll see the best performance when you use a CDN, regardless of which region your Spaces are in.

How Does This Improve Performance?

Droplets and Spaces in the same datacenter will have the least amount of latency, but if your application requires connectivity across multiple regions, choose datacenter locations connected by DigitalOcean global backbone.

Traffic between Spaces in NYC3, AMS3, SFO2 and Droplets in NYC1, NYC2, NYC3, AMS3, LON1, FRA1, TOR1, SFO1, AMS2, will all go over the global backbone. This will provide predictable and stable latency with no packet loss because these connections use our dedicated links instead of the public Internet.

How Do I Implement This?

You can choose the regions for your resources at creation time. For existing infrastructure, you can migrate your Droplets to a new region.

Traffic between Spaces in NYC3 and Droplets in NYC1, NYC2, and NYC3 go over our Northeast regional backbone; traffic between Spaces in AMS3 and Droplets in AMS2, LON1, and FRA1 go over our European regional backbone.

Handle 50x Errors Properly

When Should I Do This?

This recommendation applies any time you upload files to Spaces. You should always make sure your upload application or library correctly handles 50x errors.

How Does This Improve Performance?

With correct error handling and retry logic, your dataset will upload faster and require less human intervention. Additionally, without this functionality, any 50x errors during uploads will create gaps in your data.

How Do I Implement This?

Spaces has a very high degree of compatibility with S3, so one option is to use a well-established S3-compatible client or library for your uploads, like s3cmd.

If you’re writing your own code, make sure to implement retry logic with exponential backoff to handle 503 Slow Down responses.

Optimize Your Request Rate

When Should I Do This?

You should consider ways to optimize the number of requests you send to Spaces if:

  • You send more than 200 requests per second.
  • Your requests are being rate limited.
  • You frequently encounter 503 Slow Down responses.

As mentioned in our release notes, on 15 Feb 2018 we resolved an issue where some Spaces customers were being rate limited despite being below the threshold.

How Does This Improve Performance?

To ensure all customers receive a fair share of Spaces’ available throughput in any region, we apply rate limiting at the Space level. Making sure your uploads stay within the threshold will prevent unexpected throttling. As we continue to improve Spaces, we will also re-evaluate this threshold.

How Do I Implement This?

There are several ways to begin optimizing the number of requests you send depending on your use case.

  • If you’re uploading many small files at a high rate, consider combining small files into larger files.
  • If you’re running into throttling issues, make sure your upload application or library is handling 50x errors correctly.
  • If you plan to push more than 200 requests/second to Spaces (regularly or as part of a one-time upload), open a support ticket so we can help you prepare for the workload and avoid any temporary limits on your request rate.

Use Local or Block Storage Instead

When Should I Do This?

Not all use cases are appropriate for Spaces. Like all object stores, Spaces are best used for static, unstructured data.

You should use local or block storage if:

  • You’re storing dynamic or structured data, like low-latency key/value stores and other databases.
  • You have a high request rate for your data, like streaming video.

How Does This Improve Performance?

Using local storage or block storage will give you better performance in certain cases because the underlying hardware devices provide low-latency I/O. You can visit Object Storage vs. Block Storage Services to learn more.

How Do I Implement This?

You can get started with Volumes, or learn more about Linux storage concepts and terminology first.

If you need a database solution, you can look into a Redis or Cassandra cluster.