How To Mirror Local and Remote Directories on a VPS with lsyncd

What is lsyncd?

When managing a web server or an application, there are many situations that require synchronization between directories. While a number of tools can help you accomplish this, lsyncd is a strong candidate.

This article will discuss how to configure lsyncd to mirror changes between directories on a single machine and how to mirror between remote hosts.

For this guide, we will be using an Ubuntu 12.04 VPS, but any modern distribution should function in a similar way.

How to Install lsyncd

Fortunately, Ubuntu includes lsyncd in its default repositories.

We can install lsyncd with the following commands:

sudo apt-get update
sudo apt-get install lsyncd

This will install lsync, but it will not provide you with a default configuration. We will create one later in the article.

How to Find lsyncd Configuration Examples

While lsyncd does not provide a configuration file by default, it does include some examples that we can look at to get ideas.

See the examples by checking out the files in the "/usr/share/doc/lsyncd/examples" directory:

cd /usr/share/doc/lsyncd/examples
ls

lbash.lua  lgforce.lua      lpostcmd.lua  lrsyncssh.lua
lecho.lua  limagemagic.lua  lrsync.lua

You can look at these text files to get a sense of what can be done with configuration.

We can check out one of the more basic configurations by opening the "lrsync.lua" file:

sudo nano 

----
-- User configuration file for lsyncd.
--
-- Simple example for default rsync.
--
settings = {
        statusFile = "/tmp/lsyncd.stat",
        statusIntervall = 1,
}

sync{
        default.rsync,
        source="src",
        target="trg",
}

The lines beginning with two dashes (--) are comments. They are not interpreted by lsyncd.

The lsync configuration file is written in the Lua programming language. You can learn more about Lua here.

Setting Up the Environment

We will sync two local directories in our first example. Let's create them with the following commands:

sudo mkdir -p /source/to/copy
sudo mkdir /dest

We will then add some files to the first directory so that we can check if the sync is operating correctly:

cd /source/to/copy
sudo touch file{1..100}

The above commands create 100 files in the "/source/to/copy" directory.

Additionally, we can create a log directory and some files for lsyncd to use:

sudo mkdir /var/log/lsyncd
touch /var/log/lsyncd/lsyncd.{log,status}

Next, we can create the lsyncd configuration directory:

sudo mkdir /etc/lsyncd

We will create a configuration file inside of this directory called "lsyncd.conf.lua", with nano:

sudo nano /etc/lsyncd/lsyncd.conf.lua

How to Sync Two Local Folders with lsyncd

The configuration file we are making will sync the "/source/to/copy" directory to the "/dest" directory.

Configuring the Global Section

The general settings are all configured in a section called "settings". Our section will contain a few basic settings:

settings = {
	logfile = "/var/log/lsyncd/lsyncd.log",
	statusFile = "/var/log/lsyncd/lsyncd.status"
}

These options tell lsyncd to use the files that we created earlier.

Configuring the Sync Section

The next section specifies how you want the sync operations to be performed. Since this is a local transfer, we will use regular rsync to complete the transfer.

This configuration is defined by the "default.rsync" option. This configuration means that lsyncd will wait 20 seconds or collect 1000 separate sync events and then call rsync with the changes that are needed.

According to the lsyncd manual, the rsync instance that is called is equivalent to the command:

rsync -ltsd --delete --include-from=- --exclude=* SOURCE TARGET

This means that rsync operates in the following way:

• -l: copy symbolic links
• -t: copy modification time
• -s: no space-splitting; wildcard characters only
• -d: transfer directories without recursing

The only thing we need to do to implement these changes is to specify the sync type and the source and target directories. The sync section will look like this:

sync {
	default.rsync,
	source = "/source/to/copy",
	target = "/dest"
}

If you want to add options to rsync to modify its behavior, you can do so by passing the "rsyncOpts" variable an array containing comma-separated strings, each representing an rsync option:

sync {
	default.rsync,
	source = "/source/to/copy",
	target = "/dest",
	rsyncOpts = {"rsync option1", "rsync option2", "rsync option3"}
}

We now have enough of a configuration to test our setup. Save and close the file.

Testing Local Syncing

Let's go to the target directory and verify that there are no files in the "/dest" directory at this point:

cd /dest
ls

The last command should return no output because the "/dest" directory should be empty.

We can start the lsyncd service by issuing the following command:

sudo service lsyncd start

Check the directory again with ls:

ls

file1    file18  file27  file36  file45  file54  file63  file72  file81  file90
file10   file19  file28  file37  file46  file55  file64  file73  file82  file91
file100  file2   file29  file38  file47  file56  file65  file74  file83  file92
file11   file20  file3   file39  file48  file57  file66  file75  file84  file93
file12   file21  file30  file4   file49  file58  file67  file76  file85  file94
file13   file22  file31  file40  file5   file59  file68  file77  file86  file95
file14   file23  file32  file41  file50  file6   file69  file78  file87  file96
file15   file24  file33  file42  file51  file60  file7   file79  file88  file97
file16   file25  file34  file43  file52  file61  file70  file8   file89  file98
file17   file26  file35  file44  file53  file62  file71  file80  file9   file99

We can see that all of the files are instantly synced.

If we add some files to the source directory, we will experience the 20 second delay that we mentioned when discussing "default.rsync":

sudo mkdir /source/to/copy/hello{1..100}
ls

We won't see the files initially, but after the timeout is reached, rsync will sync the files. Check again by reissuing the "ls" command.

How to Configure Remote Syncing with lsyncd

With a few changes to our configuration file, we can configure remote syncing.

First we need to be able to sign into the mirroring machine from the original machine through password-less ssh.

How to Log Into the Remote Machine with SSH Keys

If you have not already done so, generate ssh keys on your machine with lsyncd by following this guide.

Make a key pair for your root user, since the lsyncd service will run as root. You can then copy the key file to the remote mirror machine with the following commands:

sudo su
ssh-copy-id remote_mirror_ip_address

Your key file should now allow you to log into the remote mirroring server as the root user.

Try it now so to test that it works and to create a destination directory on the remote host:

ssh remote_mirror_ip_address

We will create a directory called "/remotesync" to act as our target directory:

mkdir /remotesync

Exit out of the remote session and the local root session by typing "exit" twice:

exit
exit

How to Configure lsyncd to Mirror Remotely

We can open the lsyncd configuration file again with the following command:

sudo nano /etc/lsyncd/lsyncd.conf.lua

settings = {
        logfile = "/var/log/lsyncd/lsyncd.log",
        statusFile = "/var/log/lsyncd/lsyncd.status"
}

sync {
        default.rsync,
        source = "/source/to/copy",
        target = "/dest"
}

We will only need to make changes within the "sync" section.

We will change "default.rsync" to "default.rsyncssh" to enable rsync over ssh, and we will replace the "target" variable with the "host" and "targetdir" variables:

settings = {
        logfile = "/var/log/lsyncd/lsyncd.log",
        statusFile = "/var/log/lsyncd/lsyncd.status"
}

sync {
        default.rsyncssh,
        source = "/source/to/copy",
        host = "remote_mirror_ip_address",
        targetdir = "/remotesync"
}

Save the file and exit.

Testing Remote Syncing

We can now restart the lsyncd service by issuing the following command:

sudo service lsyncd restart

If we ssh into our remote mirror, we should be able to see the changes in the remote "/remotesync" directory:

sudo su
ssh remote_mirror_ip_address
ls /remotesync

You should see all of the files that we added to the local "/source/to/copy" directory.

Taking it Further

The lsyncd service is a great way to synchronize files across directories or systems. Due to its Lua-based configuration files, it can be very powerful and flexible. The Lua documentation (linked earlier), and the lsync documentation are great resources that will allow you to develop more complex syncing operations.

Check out some of the other examples in "/usr/share/doc/lsyncd/examples" directory for more ideas.