By radiance
My goal is to recursively download a directory from DigitalOcean spaces to local machine (nothing fancy). If I attempt to use AWS CLI for the job, aws s3 commands download only first 1000 files from any directory that contains more than 1000 files.
To reproduce:
- Create directory
test1with 1007 files locally - Upload this directory to spaces:
aws s3 cp --recursive --endpoint=https://reg1.digitaloceanspaces.com s3://mybucket/test1/ ./test1/ - Upload works as expected (uploads all 1007 files)
- Try to download this directory locally:
aws s3 cp --recursive --endpoint=https://reg1.digitaloceanspaces.com s3://mybucket/test1/ ./download1/ - Downloaded directory contains only 1000 files
Adding argument --page-size=500 to aws s3 cp command, downloads only first 500 files, so it downloads only the first page. Reproduced on Linux, macOS, multiple AWS CLI, multiple Python versions.
Is downloading entire bucket or directory not possible with AWS CLI? This seems like a very basic feature that should just work.
Become a contributor for community
Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.
DigitalOcean Documentation
Full documentation for every DigitalOcean product.
Resources for startups and AI-native businesses
The Wave has everything you need to know about building a business, from raising funding to marketing your product.
The developer cloud
Scale up as you grow — whether you're running one virtual machine or ten thousand.
Start building today
From GPU-powered inference and Kubernetes to managed databases and storage, get everything you need to build, scale, and deploy intelligent applications.
