Question

Uploading to DO Spaces via Boto3 prepends the bucket name to file keys

Say I have a DO Spaces bucket called my-test-bucket.

When I upload to Spaces using the Python Boto3 library’s upload_fileobj method where Bucket=“my-test-bucket” and Key=“projects/1/data/1/testfile.txt” then the resulting full file path that actually gets created/shown in the Spaces dashboard is /my-test-bucket/my-test-bucket/projects/1/data/1/testfile.txt

Here is the URL that would be shown when I find the item from the DO Spaces dashboard: https://my-test-bucket.sfo3.digitaloceanspaces.com/my-test-bucket/projects/1/datasets/1/testfile.txt

This is breaking my subsequent list_files_v2 calls later on that are passing Bucket=‘my-test-project’ and Prefix=‘projects/1/data/1/’ as arguments.

For additional context, I have double-checked my bucket name and file key arguments to ensure I’m not accidentally putting the bucket name into the file key. I also do not have any periods in my bucket name.

Would appreciate any assistance here as I though I was going crazy until I saw someone having a similar issue here.

I have tested my application using the exact same Boto3 commands except running against MinIO (self-hosted S3 compatible storage server) instead of DO Spaces, and I am getting the desired behavior without the prepended bucket name, which leads me to believe that the issue is with DO Spaces rather than the commands my app is issuing or the Boto3 library.

I’m not sure if the issue is with the uploads, or with the file listing. It’s almost as if the list_files_v2 is expecting the bucket_name in both the bucket argument as well as part of the prefix to the file.


Submit an answer


This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

Sign In or Sign Up to Answer

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

Bobby Iliev
Site Moderator
Site Moderator badge
July 6, 2024

Hey!

I’ve just tested this with the following simple script and it seems to have worked as expected:

import os
import boto3
from dotenv import load_dotenv

# Load environment variables from .env file
load_dotenv()

# Set your credentials and region
spaces_key = os.getenv('DO_SPACES_KEY')
spaces_secret = os.getenv('DO_SPACES_SECRET')
spaces_region = os.getenv('DO_SPACES_REGION')
spaces_name = os.getenv('DO_SPACES_NAME')

# Create a session and client
session = boto3.session.Session()
client = session.client('s3',
                        region_name=spaces_region,
                        endpoint_url=f'https://{spaces_region}.digitaloceanspaces.com',
                        aws_access_key_id=spaces_key,
                        aws_secret_access_key=spaces_secret)

def upload_file(file_name, bucket, object_name=None):
    """Upload a file to an S3 bucket

    :param file_name: File to upload
    :param bucket: Bucket to upload to
    :param object_name: S3 object name. If not specified then file_name is used
    :return: True if file was uploaded, else False
    """
    if object_name is None:
        object_name = file_name

    # Upload the file
    try:
        client.upload_file(file_name, bucket, object_name)
    except Exception as e:
        print(f"Error uploading file: {e}")
        return False
    return True

# Example usage
file_name = 'test.txt'
bucket_name = spaces_name
object_name = 'destination/file.txt'

# Upload the file
if upload_file(file_name, bucket_name, object_name):
    print(f"File {file_name} uploaded successfully to {bucket_name}/{object_name}")
else:
    print(f"Failed to upload file {file_name}")

The file was uploaded to the destination directory without having the bucket name prepended two times:

Would you mind sharing an example code snippet that you are using so I could further look into this?

- Bobby

Try DigitalOcean for free

Click below to sign up and get $200 of credit to try our products over 60 days!

Sign up

Become a contributor for community

Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.

DigitalOcean Documentation

Full documentation for every DigitalOcean product.

Resources for startups and SMBs

The Wave has everything you need to know about building a business, from raising funding to marketing your product.

Get our newsletter

Stay up to date by signing up for DigitalOcean’s Infrastructure as a Newsletter.

New accounts only. By submitting your email you agree to our Privacy Policy

The developer cloud

Scale up as you grow — whether you're running one virtual machine or ten thousand.

Get started for free

Sign up and get $200 in credit for your first 60 days with DigitalOcean.*

*This promotional offer applies to new accounts only.