We hope you find this tutorial helpful. In addition to guides like this one, we provide simple cloud infrastructure for developers. Learn more →

How To Configure s3cmd 2.x To Manage DigitalOcean Spaces

PostedSeptember 20, 2017 6.5k views Object Storage DigitalOcean

Introduction

Welcome to the DigitalOcean Spaces. In this article, we'll show you how to install, configure, and manage DigitalOcean Spaces with the command-line tool s3cmd.

Prerequisites

To use this 3rd party client, you will need:

When you have completed the prerequisites, you're ready to follow along.

Verify the Version

We'll begin by confirming our version of s3cmd: with the following command:

  • s3cmd --version

Output:

s3cmd version 2.0.0+

By verifying that we're running the same version, we can expect that the output of our commands will match the diretions that follow.

Choose the Configuration File

By default, s3cmd stores its configuration file, .s3cfg, in the home directory of the user who runs the configuration command. This is a plain text file of key/value pairs which can be edited directly once it has been created. You can choose between setting up DigitalOcean as the default configuration or creating a different configuration file:

Option 1: Make DigitalOcean the default connection

If DigitalOcean is the main or only provider you'll connect to with s3cmd, configure it in the default ~/.s3cfg file with the following command:

s3cmd --configure

By choosing this option, you won't have to specify the configuration file each time you run a command.

Option 2: Create an explicit configuration file for DigitalOcean

If you're already using s3cmd with another service, you may want to create an alternate configuration file, which you can do by adding the -c flag and supplying a filename. The configuration file will be created in the directory where you issue the command, so specify the path if you want it created elsewhere.

In this example, we'll place a config file named nyc3in our home directory:

  • s3cmd --configure -c ~/nyc3

Important: To use this configuration file, it must be explicitly provided at the end of each command by appending -c ~/nyc3.

Configure s3cmd

Whether you use the default configuration file or specify your own, when you issue the configuration command, it will launch an interactive multi-step configuration script. In this section, we'll go through each of the steps in the configuration process.

Enter Access Keys

The script begins by asking for an Access Key and Secret Key. If you don't already have keys, you can generate a set for s3cmd by visiting the Control Panel's API page.

We'll enter our keys, then accept US for the Default Region since the region information isn't relevant to DigitalOcean.

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.
Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key []: EXAMPLE7UQOTHDTF3GK4
Secret Key []: b8e1ec97b97bff326955375c5example
Default Region [US]:

Note: If you prefer, you can use the environment variables AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY to store a set of keys.

Set the Endpoint

Next, we'll enter the early access DigitalOcean endpoint, nyc3.digitaloceanspaces.com

Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.amazonaws.com]: nyc3.digitaloceanspaces.com

Since Spaces supports DNS-based buckets, at the next prompt, we'll supply the bucket value in the required format:

%(bucket)s.nyc3.digitaloceanspaces.com

Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars c
an be used if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket []: %(bucket)s.nyc3.digitaloceanspaces.com

Optional: Set an encryption password

Next, we're asked to supply an encryption password. Unlike HTTPS, which protects file only while in transit, GPG encryption prevents others from reading files while they are stored on DigitalOcean as well as in transit. Setting a password now won't cause objects to be automatically encrypted; that's done by adding a -e flag to the put command when you copy the object to a Space.

We'll enter a password so it will be available in the event we want to use encryption.:

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: secure_password
Path to GPG program [/usr/bin/gpg]:

Connect via HTTPS

Next, we're prompted to connect via HTTPS, which protects data from being read while it is in transit. DigitalOcean Spaces do not support unencrypted transfer, so we'll press ENTER to accept the default, YES:

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]: Yes'

Optional: Set a Proxy Server

If your network requires you to use an HTTP Proxy server, enter its IP address or domain name without the protocol, e.g. 203.0.113.1 or proxy.example.com Since we aren't using an HTTP Proxy server, we'll leave this question blank and press ENTER:

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name:

Confirm and test settings

After the prompt for the HTTP Proxy server name, the configuration script presents a summary of the values it will use, followed by the opportunity to test them:

New settings:
 Access Key: EXAMPLES7UQOTHDTF3GK4
 Secret Key: b8e1ec97b97bff326955375c5example
 Default Region: US
 S3 Endpoint: nyc3.digitaloceanspaces.com
 DNS-style bucket+hostname:port template for accessing a bucket: %(bucket)s.n
yc3.digitaloceanspaces.com
 Encryption password: secure_password
 Path to GPG program: /usr/bin/gpg
 Use HTTPS protocol: True
 HTTP Proxy server name:
 HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] Y

Save the settings

When the test completes successfully, enter Y to save the settings:

Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Success. Encryption and decryption worked fine :-)

Save settings? [y/N]  Y

If the test fails or you choose N you'll have the opportunity to retry the configuration. Once you save the configuration, you'll receive confirmation of its location:

  • Configuration saved to '/home/sammy/nyc3'

The test step confirms that we can connect to our DigitalOcean Spaces.

Conclusion

In this tutorial we've configured and tested the s3cmd settings.

16 Comments

Creative Commons License