Question

Uploading to DO Spaces via Boto3 prepends the bucket name to file keys

Say I have a DO Spaces bucket called my-test-bucket.

When I upload to Spaces using the Python Boto3 library’s upload_fileobj method where Bucket=“my-test-bucket” and Key=“projects/1/data/1/testfile.txt” then the resulting full file path that actually gets created/shown in the Spaces dashboard is /my-test-bucket/my-test-bucket/projects/1/data/1/testfile.txt

Here is the URL that would be shown when I find the item from the DO Spaces dashboard: https://my-test-bucket.sfo3.digitaloceanspaces.com/my-test-bucket/projects/1/datasets/1/testfile.txt

This is breaking my subsequent list_files_v2 calls later on that are passing Bucket=‘my-test-project’ and Prefix=‘projects/1/data/1/’ as arguments.

For additional context, I have double-checked my bucket name and file key arguments to ensure I’m not accidentally putting the bucket name into the file key. I also do not have any periods in my bucket name.

Would appreciate any assistance here as I though I was going crazy until I saw someone having a similar issue here.

I have tested my application using the exact same Boto3 commands except running against MinIO (self-hosted S3 compatible storage server) instead of DO Spaces, and I am getting the desired behavior without the prepended bucket name, which leads me to believe that the issue is with DO Spaces rather than the commands my app is issuing or the Boto3 library.

I’m not sure if the issue is with the uploads, or with the file listing. It’s almost as if the list_files_v2 is expecting the bucket_name in both the bucket argument as well as part of the prefix to the file.


Submit an answer


This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

Sign In or Sign Up to Answer

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

Bobby Iliev
Site Moderator
Site Moderator badge
July 6, 2024

Hey!

I’ve just tested this with the following simple script and it seems to have worked as expected:

import os
import boto3
from dotenv import load_dotenv

# Load environment variables from .env file
load_dotenv()

# Set your credentials and region
spaces_key = os.getenv('DO_SPACES_KEY')
spaces_secret = os.getenv('DO_SPACES_SECRET')
spaces_region = os.getenv('DO_SPACES_REGION')
spaces_name = os.getenv('DO_SPACES_NAME')

# Create a session and client
session = boto3.session.Session()
client = session.client('s3',
                        region_name=spaces_region,
                        endpoint_url=f'https://{spaces_region}.digitaloceanspaces.com',
                        aws_access_key_id=spaces_key,
                        aws_secret_access_key=spaces_secret)

def upload_file(file_name, bucket, object_name=None):
    """Upload a file to an S3 bucket

    :param file_name: File to upload
    :param bucket: Bucket to upload to
    :param object_name: S3 object name. If not specified then file_name is used
    :return: True if file was uploaded, else False
    """
    if object_name is None:
        object_name = file_name

    # Upload the file
    try:
        client.upload_file(file_name, bucket, object_name)
    except Exception as e:
        print(f"Error uploading file: {e}")
        return False
    return True

# Example usage
file_name = 'test.txt'
bucket_name = spaces_name
object_name = 'destination/file.txt'

# Upload the file
if upload_file(file_name, bucket_name, object_name):
    print(f"File {file_name} uploaded successfully to {bucket_name}/{object_name}")
else:
    print(f"Failed to upload file {file_name}")

The file was uploaded to the destination directory without having the bucket name prepended two times:

Would you mind sharing an example code snippet that you are using so I could further look into this?

- Bobby

Try DigitalOcean for free

Click below to sign up and get $200 of credit to try our products over 60 days!

Sign up

Featured on Community

Get our biweekly newsletter

Sign up for Infrastructure as a Newsletter.

Hollie's Hub for Good

Working on improving health and education, reducing inequality, and spurring economic growth? We'd like to help.

Become a contributor

Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.

Welcome to the developer cloud

DigitalOcean makes it simple to launch in the cloud and scale up as you grow — whether you're running one virtual machine or ten thousand.

Learn more
DigitalOcean Cloud Control Panel