GlusterFS Geo-Replication


I have three nodes: nginx01-03 (all running Ubuntu 14.04). The nodes nginx01/02 currently reside in NYC3 and have GlusterFS (v3.6.5) replication working between them without issue via the private network. I have modified the /etc/hosts file of all three nodes to point to the respective parties IP address that’s necessary (in this case: 01 and 02 see the private IP of each other and the public IP of 03 whereas 03 sees the public IP of 01 and 02). They’ve been given the shortname of nginx01-03 respectively.


I was able to establish replication between two nodes in the same DC (NYC3) over the private network without issue; however I’m trying to utilize the geo-replication feature of GlusterFS with my NYC3 node(s) and a LON node.

I’ve attempted to use instructions found here:

However there are numerous issues that I’ve encountered:

  1. the secret.pem file did not allow for password-less SSH. I had to modify the ~/.ssh/config file on the master node to specifically point to that file by using the IdentityFile option.

  2. the command to initiate the volume on the master node to the slave node failed:

root@nginx01:~# gluster volume geo-replication primaryvol nginx03:/gluster/ start
Staging failed on localhost. Please check the log file for more details.
geo-replication command failed

I have no idea where the “log file” in question is; however I utilized the --log-file= option to specify one. It proved less than helpful as the only information I received was the following two lines:

root@nginx01:~# tail /var/log/gluster_error
[2015-09-06 12:51:34.949782] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now
[2015-09-06 12:51:35.205223] I [input.c:36:cli_batch] 0-: Exiting with: -1

Anyone have success setting this up or using geo-replication in general? Any additional information you need just let me know and I’ll gladly provide!

Show comments

Submit an answer

This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

Sign In or Sign Up to Answer

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

Want to learn more? Join the DigitalOcean Community!

Join our DigitalOcean community of over a million developers for free! Get help and share knowledge in Q&A, subscribe to topics of interest, and get courses and tools that will help you grow as a developer and scale your project or business.

This question was answered by @rgb000:

Hello. I succeed to setup geo-replication like you want on Ubuntu. Let say, you have one LAN volume nginx01-02 with name ‘lan-volume’ - this is Master volume. Slave volume is nginx3 with name ‘geo-volume’. Now your settings should match next conditions:

  1. Size of Master volume should be less then Slave
  2. There should be passwordless ssh between all three nodes
  3. You should fix gsyncd bug on Ubuntu

Condition 1.

root@client1:~# df -h
Filesystem                       Size  Used Avail Use% Mounted on
nginx01_02:/geo-volume          158G  5.0G  145G   4% /mnt/geo_volume
nginx3:/lan-volume              79G  2.5G   73G   4% /mnt/lan_volume

Total size of Master(lan-volume) is less then Slave(geo-volume).

Condition 2. Make passwordless ssh for root. Generate authorized and private key:

root@nginx01:~# ssh-keygen -t rsa -f /root/.ssh/id_rsa -q -N ""
root@nginx01:~# mv /root/.ssh/ /root/.ssh/authorized_keys
root@nginx01:~# ls /root/.ssh/
authorized_keys  id_rsa

Now sync folder to all nodes.

root@nginx01:~# rsync -avP /root/.ssh/ nginx02:/root/.ssh/
root@nginx01:~# rsync -avP /root/.ssh/ nginx03:/root/.ssh/

Now connect to each of them to add to known hosts and check connection.

root@nginx01:~# ssh nginx02
The authenticity of host 'nginx02 (' can't be established.
ECDSA key fingerprint is 11:5d:55:29:8a:77:d8:08:b4:00:9b:a3:61:93:fe:e5.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'nginx01' (ECDSA) to the list of known hosts.

root@nginx01:~# ssh nginx03
The authenticity of host 'nginx02 (' can't be established.
ECDSA key fingerprint is 31:yd:75:89:85:54:h6:65:b4:00:9b:a3:55:45:e5:23.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'nginx03' (ECDSA) to the list of known hosts.

Condition 3. Geo-replication is using ‘gsyncd’ and it should be located in /usr/libexec/glusterfs/. On Ubuntu this pass doesn’t exist. So you need to create it and make symbolic link to existing one all three nodes

root@nginx01:~# mkdir -p /usr/libexec/glusterfs/     
root@nginx01:~# ln -s /usr/lib/x86_64-linux-gnu/glusterfs/gsyncd /usr/libexec/glusterfs/gsyncd 
root@nginx02:~# mkdir -p /usr/libexec/glusterfs/     
root@nginx02:~# ln -s /usr/lib/x86_64-linux-gnu/glusterfs/gsyncd /usr/libexec/glusterfs/gsyncd 
root@nginx03:~# mkdir -p /usr/libexec/glusterfs/     
root@nginx03:~# ln -s /usr/lib/x86_64-linux-gnu/glusterfs/gsyncd /usr/libexec/glusterfs/gsyncd 

Now you just need to login to Master, setup geo-volume and start geo-replication as in instruction you give link to.

View the original comment