Best method for uploading large (9GB) file?

August 14, 2014 4k views

I'm trying to upload a large file, 9.5GB, to my droplet. It's in a directory with many other files that I need on the droplet, so I tried rsync:

rsync -az --progress grassdata/trichastll username@dr.op.let.ip:grassdata/

This appeared to work, but the pipe is broken after a while - 30 minutes to an hour, when the upload will take ca. 4 hours to complete. Restarting rsync did not restart with the partially uploaded file, but instead started to reload this particular large file completely.

Now I'm trying sftp, with compressionLevel=9 and TCPKeepAlive=yes. So far so good, but it will take another 3 hours to complete, assuming it works (after which I'll return to rsync to get the rest of my smaller files on board). In the meantime, I wonder if there is another option for uploading large files reliably that I should use instead?

specs: Debian 7/64bit on 32GB RAM server

EDIT: just found out about rsync --partial. I'll try that next time for sure. Still interested in alternatives!


1 Answer

I think you're looking for the --partial flag for rsync:

By default, rsync will delete any partially transferred file if the transfer is interrupted. In some circumstances it is more desirable to keep partially transferred files. Using the --partial option tells rsync to keep the partial file which should make a subsequent transfer of the rest of the file much faster.

By default, rsync will delete the temporary file it creates if a transfer fails.

  • Thanks Andrew!

    I had assumed the default behaviour was to keep partial transfers, so I'll use --partial from now on, at least for big files.

Have another answer? Share your knowledge.