Question

DigitalOcean Spaces: cannot download directory with more than 1000 files

My goal is to recursively download a directory from DigitalOcean spaces to local machine (nothing fancy). If I attempt to use AWS CLI for the job, aws s3 commands download only first 1000 files from any directory that contains more than 1000 files.

To reproduce:

  • Create directory test1 with 1007 files locally
  • Upload this directory to spaces: aws s3 cp --recursive --endpoint=https://reg1.digitaloceanspaces.com s3://mybucket/test1/ ./test1/
  • Upload works as expected (uploads all 1007 files)
  • Try to download this directory locally: aws s3 cp --recursive --endpoint=https://reg1.digitaloceanspaces.com s3://mybucket/test1/ ./download1/
  • Downloaded directory contains only 1000 files

Adding argument --page-size=500 to aws s3 cp command, downloads only first 500 files, so it downloads only the first page. Reproduced on Linux, macOS, multiple AWS CLI, multiple Python versions.

Is downloading entire bucket or directory not possible with AWS CLI? This seems like a very basic feature that should just work.

Show comments

Submit an answer

This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

Sign In or Sign Up to Answer

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

Want to learn more? Join the DigitalOcean Community!

Join our DigitalOcean community of over a million developers for free! Get help and share knowledge in Q&A, subscribe to topics of interest, and get courses and tools that will help you grow as a developer and scale your project or business.

Same here… wtf is going on? how to sync files?

Running into the same issue, this is very frustrating since all s3 operations will be silently capped at 1000 files max.

I assume this is mentioned as a ‘known issues’ at https://www.digitalocean.com/docs/spaces/

In the API, list-objects-v2 pagination does not work.

It basically means do not use DO spaces for anything serious