Tutorial

How To Use Rsync to Sync Local and Remote Directories

Updated on January 28, 2022
English
How To Use Rsync to Sync Local and Remote Directories

Introduction

Rsync, which stands for remote sync, is a remote and local file synchronization tool. It uses an algorithm to minimize the amount of data copied by only moving the portions of files that have changed.

In this tutorial, we’ll define Rsync, review the syntax when using rsync, explain how to use Rsync to sync with a remote system, and other options available to you.

Deploy your frontend applications from GitHub using DigitalOcean App Platform. Let DigitalOcean focus on scaling your app.

Prerequisites

In order to practice using rsync to sync files between a local and remote system, you will need two machines to act as your local computer and your remote machine, respectively. These two machines could be virtual private servers, virtual machines, containers, or personal computers as long as they’ve been properly configured.

If you plan to follow this guide using servers, it would be prudent to set them up with administrative users and to configure a firewall on each of them. To set up these servers, follow our Initial Server Setup Guide.

Regardless of what types of machines you use to follow this tutorial, you will need to have created SSH keys on both of them. Then, copy each server’s public key to the other server’s authorized_keys file as outlined in Step 2 of that guide.

This guide was validated on machines running Ubuntu 20.04, although it should generally work with any computers running a Linux-based operating system that have rsync installed.

Defining Rsync

Rsync is a very flexible network-enabled syncing tool. Due to its ubiquity on Linux and Unix-like systems and its popularity as a tool for system scripts, it’s included on most Linux distributions by default.

Understanding Rsync Syntax

The syntax for rsync operates similar to other tools, such as ssh, scp, and cp.

First, change into your home directory by running the following command:

  1. cd ~

Then create a test directory:

  1. mkdir dir1

Create another test directory:

  1. mkdir dir2

Now add some test files:

  1. touch dir1/file{1..100}

There’s now a directory called dir1 with 100 empty files in it. Confirm by listing out the files:

  1. ls dir1
Output
file1 file18 file27 file36 file45 file54 file63 file72 file81 file90 file10 file19 file28 file37 file46 file55 file64 file73 file82 file91 file100 file2 file29 file38 file47 file56 file65 file74 file83 file92 file11 file20 file3 file39 file48 file57 file66 file75 file84 file93 file12 file21 file30 file4 file49 file58 file67 file76 file85 file94 file13 file22 file31 file40 file5 file59 file68 file77 file86 file95 file14 file23 file32 file41 file50 file6 file69 file78 file87 file96 file15 file24 file33 file42 file51 file60 file7 file79 file88 file97 file16 file25 file34 file43 file52 file61 file70 file8 file89 file98 file17 file26 file35 file44 file53 file62 file71 file80 file9 file99

You also have an empty directory called dir2. To sync the contents of dir1 to dir2 on the same system, you will run rsync and use the -r flag, which stands for “recursive” and is necessary for directory syncing:

  1. rsync -r dir1/ dir2

Another option is to use the -a flag, which is a combination flag and stands for “archive”. This flag syncs recursively and preserves symbolic links, special and device files, modification times, groups, owners, and permissions. It’s more commonly used than -r and is the recommended flag to use. Run the same command as the previous example, this time using the -a flag:

  1. rsync -a dir1/ dir2

Please note that there is a trailing slash (/) at the end of the first argument in the syntax of the the previous two commands and highlighted here:

  1. rsync -a dir1/ dir2

This trailing slash signifies the contents of dir1. Without the trailing slash, dir1, including the directory, would be placed within dir2. The outcome would create a hierarchy like the following:

~/dir2/dir1/[files]

Another tip is to double-check your arguments before executing an rsync command. Rsync provides a method for doing this by passing the -n or --dry-run options. The -v flag, which means “verbose”, is also necessary to get the appropriate output. You’ll combine the a, n, and v flags in the following command:

  1. rsync -anv dir1/ dir2
Output
sending incremental file list ./ file1 file10 file100 file11 file12 file13 file14 file15 file16 file17 file18 . . .

Now compare that output to the one you receive when removing the trailing slash, as in the following:

  1. rsync -anv dir1 dir2
Output
sending incremental file list dir1/ dir1/file1 dir1/file10 dir1/file100 dir1/file11 dir1/file12 dir1/file13 dir1/file14 dir1/file15 dir1/file16 dir1/file17 dir1/file18 . . .

This output now demonstrates that the directory itself was transferred, rather than only the files within the directory.

Using Rsync to Sync with a Remote System

To use rsync to sync with a remote system, you only need SSH access configured between your local and remote machines, as well as rsync installed on both systems. Once you have SSH access verified between the two machines, you can sync the dir1 folder from the previous section to a remote machine by using the following syntax. Please note in this case, that you want to transfer the actual directory, so you’ll omit the trailing slash:

  1. rsync -a ~/dir1 username@remote_host:destination_directory

This process is called a push operation because it “pushes” a directory from the local system to a remote system. The opposite operation is pull, and is used to sync a remote directory to the local system. If the dir1 directory were on the remote system instead of your local system, the syntax would be the following:

  1. rsync -a username@remote_host:/home/username/dir1 place_to_sync_on_local_machine

Like cp and similar tools, the source is always the first argument, and the destination is always the second.

Using Other Rsync Options

Rsync provides many options for altering the default behavior of the utility, such as the flag options you learned about in the previous section.

If you’re transferring files that have not already been compressed, like text files, you can reduce the network transfer by adding compression with the -z option:

  1. rsync -az source destination

The -P flag is also helpful. It combines the flags --progress and --partial. This first flag provides a progress bar for the transfers, and the second flag allows you to resume interrupted transfers:

  1. rsync -azP source destination
Output
sending incremental file list created directory destination source/ source/file1 0 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=99/101) sourcefile10 0 100% 0.00kB/s 0:00:00 (xfr#2, to-chk=98/101) source/file100 0 100% 0.00kB/s 0:00:00 (xfr#3, to-chk=97/101) source/file11 0 100% 0.00kB/s 0:00:00 (xfr#4, to-chk=96/101) source/file12 0 100% 0.00kB/s 0:00:00 (xfr#5, to-chk=95/101) . . .

If you run the command again, you’ll receive a shortened output since no changes have been made. This illustrates Rsync’s ability to use modification times to determine if changes have been made:

  1. rsync -azP source destination
Output
sending incremental file list sent 818 bytes received 12 bytes 1660.00 bytes/sec total size is 0 speedup is 0.00

Say you were to update the modification time on some of the files with a command like the following:

  1. touch dir1/file{1..10}

Then, if you were to run rsync with -azP again, you’ll notice in the output how Rsync intelligently re-copies only the changed files:

  1. rsync -azP source destination
Output
sending incremental file list file1 0 100% 0.00kB/s 0:00:00 (xfer#1, to-check=99/101) file10 0 100% 0.00kB/s 0:00:00 (xfer#2, to-check=98/101) file2 0 100% 0.00kB/s 0:00:00 (xfer#3, to-check=87/101) file3 0 100% 0.00kB/s 0:00:00 (xfer#4, to-check=76/101) . . .

In order to keep two directories truly in sync, it’s necessary to delete files from the destination directory if they are removed from the source. By default, rsync does not delete anything from the destination directory.

You can change this behavior with the --delete option. Before using this option, you can use -n, the --dry-run option, to perform a test to prevent unwanted data loss:

  1. rsync -an --delete source destination

If you prefer to exclude certain files or directories located inside a directory you are syncing, you can do so by specifying them in a comma-separated list following the --exclude= option:

  1. rsync -a --exclude=pattern_to_exclude source destination

If you have a specified pattern to exclude, you can override that exclusion for files that match a different pattern by using the --include= option:

  1. rsync -a --exclude=pattern_to_exclude --include=pattern_to_include source destination

Finally, Rsync’s --backup option can be used to store backups of important files. It’s used in conjunction with the --backup-dir option, which specifies the directory where the backup files should be stored:

  1. rsync -a --delete --backup --backup-dir=/path/to/backups /path/to/source destination

Conclusion

Rsync can streamline file transfers over networked connections and add robustness to local directory syncing. The flexibility of Rsync makes it a good option for many different file-level operations.

A mastery of Rsync allows you to design complex backup operations and obtain fine-grained control over how and what is transferred.

Need highly available block storage? Attach secure, scalable NVMe- and SSD-based Volumes Block Storage to your DigitalOcean virtual machine in seconds. We’ll make sure your data is reliably stored and secure.

Learn more here


About the authors
Default avatar

Technical Writer

Educator and writer committed to empowering our community by providing access to the knowledge and tools for making creative ideas into a reality



Still looking for an answer?

Ask a questionSearch for more help

Was this helpful?
 
10 Comments


This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

A nice tip when you need to send files through a specific SSH port:

rsync -azP -e "ssh -p PORT_NUMBER" source destination

Another great Digital Ocean tutorial. Thanks a lot @jellingwood! The only thing I have missed is the mysterious rsyncd or rsync --daemon which initially seemed to me something the keeps running all the time watching for live file changes and it would be great to link to a folder. Could you please help me on clarifying what is that rsyncd really does?

I saw that @kamaln7 pointed to lsyncd which seemed pretty neat. However I still got that big question mark over my head when speaking about rsyncd. :)

Thanks again

PS: The way I currently work with live syncing is with fsmonitor npm with the following command:

fsmonitor rsync -azP --del --exclude '.git' ./ remove-server:folder

Is it possible to backup multiple folders are once over SSH? Locally, this command works fine:

rsync -av /{folder1, folder2} /backup

But when run over ssh, it fails. Any ideas? The paths are correct and work when run individually. I am on OS X if that matters.

I think the statement on excluding multiple files/comments by listing them separated with comma is dead wrong… doesn’t work… here is the right way Examples to Exclude Multiple Files and Directories using exclude-from (segment 6)

so use something like rsync -az --delete --exclude=.git --exclude=data source destination

What do i have do run if both servers has the same public key and the password auth is disabled ?

Kamal Nasser
DigitalOcean Employee
DigitalOcean Employee badge
August 14, 2014

@bcabalic: Try using the following command:

rsync -avP --numeric-ids --exclude='/dev' --exclude='/proc' --exclude='/sys' / root@xxx.xxx.xxx.xxx:/

This should make sure the permissions are properly set and that /dev, /proc, and /sys aren’t transferred.

I’m looking to do a complete “clone” of one droplet to another, for the purpose of moving from a larger droplet to a smaller droplet. That is, copy the entire root to root.

Can someone advise me on this command? Thanks in advance!

rsync -avP /* root@xxx.xxx.xxx.xxx:/

Kamal Nasser
DigitalOcean Employee
DigitalOcean Employee badge
July 31, 2014

@tech: You can use the --backup and --backup-dir arguments to do that.

-b, --backup make backups (see --suffix & --backup-dir)
--backup-dir=DIR make backups into hierarchy based in DIR

Example:

rsync -a --delete --backup --backup-dir=/path/to/backups /path/to/source destination 

Make sure --backup-dir is set to a directory that is outside of the tree that is being synced (source).

Dear Justin:

Using the --delete option, I would like to “store” (keep) the deleted files in a different directory I choose. How can I get it?

mm

I’m having a problem backing up some various configuration files from my droplet to my local machine. This problem is not specific to rsync, but that is my tool of choice for this purpose, so I’m asking here.

The problem is a permissions issue. The files I need to copy usually require elevated privileges, things like /etc/dovecot/*, which are owned by root. So,

$ rsync -a username@server:/etc/dovecot backups

fails, because my user has insufficient privileges. In an ssh session, I would simply use sudo to elevate my privileges. Also,

$ rsync -a root@server:/etc/dovecot backups

does not work, because I’ve disabled root login on my droplet.

What is the best way to accomplish this? So far I’ve managed to alarm my droplet’s firewall to the point where it blocked my own ip.

Try DigitalOcean for free

Click below to sign up and get $200 of credit to try our products over 60 days!

Sign up

Join the Tech Talk
Success! Thank you! Please check your email for further details.

Please complete your information!

Get our biweekly newsletter

Sign up for Infrastructure as a Newsletter.

Hollie's Hub for Good

Working on improving health and education, reducing inequality, and spurring economic growth? We'd like to help.

Become a contributor

Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.

Welcome to the developer cloud

DigitalOcean makes it simple to launch in the cloud and scale up as you grow — whether you're running one virtual machine or ten thousand.

Learn more
DigitalOcean Cloud Control Panel