I’m planning to use Space as my image host. The problem is I need to upload a lot of images to it. Currently I’m using upload_fileobj of boto3 client, which upload sequentially and it is very slow.

What is the recommend approach to upload lots of images?
Thank you.

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
1 answer

Hello,

I would recommend you to upload via s3cmd. Refer to below links, which guides more on that:

https://www.digitalocean.com/docs/spaces/resources/s3cmd/
https://www.digitalocean.com/docs/spaces/resources/s3cmd-usage/

Cheers,
Sri Charan

  • Thanks for answering,

    In my use case, I frequently upload a bunch of images, I also do some processing to the image in my Python code before uploading them, which makes boto3 is the only way to go.

    I’ve searched a lot on bulk uploading using boto3, but it seems like a dead end though.

    • Hi, @chi23

      Uploading an object to Spaces using boto3 requires that you edit the endpoint setting for the session. Here’s a quick example:

      import boto3
      
      session = boto3.session.Session()
      client = session.client('s3',
                              region_name='nyc3',
                              endpoint_url='https://nyc3.digitaloceanspaces.com',
                              aws_access_key_id='ACCESS_KEY',
                              aws_secret_access_key='SECRET_KEY')
      
      client.upload_file('/path/to/file.ext',  # Path to local file
                         'my-space',  # Name of Space
                         'file.ext')  # Name for remote file
      
      This answer was provided in this question:
      
      https://www.digitalocean.com/community/questions/how-to-upload-an-object-to-digital-ocean-spaces-using-python-boto3-library
      

      What you can do is to zip your files and upload them using boto3. You can also check the script that Mashoud1122 mentioned at the bottom of the original question.

      Hope that this helps!
      Regards,
      Alex

Submit an Answer