Get alerted when assets are down, slow, or vulnerable to SSL attacks—all free for a month. Learn more

Question

GlusterFS Geo-Replication

USEFUL INFO:

I have three nodes: nginx01-03 (all running Ubuntu 14.04). The nodes nginx01/02 currently reside in NYC3 and have GlusterFS (v3.6.5) replication working between them without issue via the private network. I have modified the /etc/hosts file of all three nodes to point to the respective parties IP address that’s necessary (in this case: 01 and 02 see the private IP of each other and the public IP of 03 whereas 03 sees the public IP of 01 and 02). They’ve been given the shortname of nginx01-03 respectively.

ISSUE:

I was able to establish replication between two nodes in the same DC (NYC3) over the private network without issue; however I’m trying to utilize the geo-replication feature of GlusterFS with my NYC3 node(s) and a LON node.

I’ve attempted to use instructions found here: http://www.jamescoyle.net/how-to/1037-synchronise-a-glusterfs-volume-to-a-remote-site-using-geo-replication

However there are numerous issues that I’ve encountered:

  1. the secret.pem file did not allow for password-less SSH. I had to modify the ~/.ssh/config file on the master node to specifically point to that file by using the IdentityFile option.

  2. the command to initiate the volume on the master node to the slave node failed:

root@nginx01:~# gluster volume geo-replication primaryvol nginx03:/gluster/ start
Staging failed on localhost. Please check the log file for more details.
geo-replication command failed

I have no idea where the “log file” in question is; however I utilized the --log-file= option to specify one. It proved less than helpful as the only information I received was the following two lines:

root@nginx01:~# tail /var/log/gluster_error
[2015-09-06 12:51:34.949782] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now
[2015-09-06 12:51:35.205223] I [input.c:36:cli_batch] 0-: Exiting with: -1

Anyone have success setting this up or using geo-replication in general? Any additional information you need just let me know and I’ll gladly provide!

Show comments
Submit an answer

This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

Sign In or Sign Up to Answer

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

Want to learn more? Join the DigitalOcean Community!

Join our DigitalOcean community of over a million developers for free! Get help and share knowledge in Q&A, subscribe to topics of interest, and get courses and tools that will help you grow as a developer and scale your project or business.

Andrew SB • June 7, 2016

This question was answered by @rgb000:

Hello. I succeed to setup geo-replication like you want on Ubuntu. Let say, you have one LAN volume nginx01-02 with name ‘lan-volume’ - this is Master volume. Slave volume is nginx3 with name ‘geo-volume’. Now your settings should match next conditions:

  1. Size of Master volume should be less then Slave
  2. There should be passwordless ssh between all three nodes
  3. You should fix gsyncd bug on Ubuntu

Condition 1.

root@client1:~# df -h
Filesystem                       Size  Used Avail Use% Mounted on
...
nginx01_02:/geo-volume          158G  5.0G  145G   4% /mnt/geo_volume
nginx3:/lan-volume              79G  2.5G   73G   4% /mnt/lan_volume
...

Total size of Master(lan-volume) is less then Slave(geo-volume).

Condition 2. Make passwordless ssh for root. Generate authorized and private key:

root@nginx01:~# ssh-keygen -t rsa -f /root/.ssh/id_rsa -q -N ""
root@nginx01:~# mv /root/.ssh/id_rsa.pub /root/.ssh/authorized_keys
root@nginx01:~# ls /root/.ssh/
authorized_keys  id_rsa

Now sync folder to all nodes.

root@nginx01:~# rsync -avP /root/.ssh/ nginx02:/root/.ssh/
root@nginx01:~# rsync -avP /root/.ssh/ nginx03:/root/.ssh/

Now connect to each of them to add to known hosts and check connection.

root@nginx01:~# ssh nginx02
The authenticity of host 'nginx02 (192.168.1.103)' can't be established.
ECDSA key fingerprint is 11:5d:55:29:8a:77:d8:08:b4:00:9b:a3:61:93:fe:e5.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'nginx01' (ECDSA) to the list of known hosts.
root@nginx02:~# 

root@nginx01:~# ssh nginx03
The authenticity of host 'nginx02 (192.168.1.103)' can't be established.
ECDSA key fingerprint is 31:yd:75:89:85:54:h6:65:b4:00:9b:a3:55:45:e5:23.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'nginx03' (ECDSA) to the list of known hosts.
root@nginx03:~#

Condition 3. Geo-replication is using ‘gsyncd’ and it should be located in /usr/libexec/glusterfs/. On Ubuntu this pass doesn’t exist. So you need to create it and make symbolic link to existing one all three nodes

root@nginx01:~# mkdir -p /usr/libexec/glusterfs/     
root@nginx01:~# ln -s /usr/lib/x86_64-linux-gnu/glusterfs/gsyncd /usr/libexec/glusterfs/gsyncd 
root@nginx02:~# mkdir -p /usr/libexec/glusterfs/     
root@nginx02:~# ln -s /usr/lib/x86_64-linux-gnu/glusterfs/gsyncd /usr/libexec/glusterfs/gsyncd 
root@nginx03:~# mkdir -p /usr/libexec/glusterfs/     
root@nginx03:~# ln -s /usr/lib/x86_64-linux-gnu/glusterfs/gsyncd /usr/libexec/glusterfs/gsyncd

Now you just need to login to Master, setup geo-volume and start geo-replication as in instruction you give link to.

View the original comment

Try DigitalOcean for free

Click below to sign up and get $200 of credit to try our products over 60 days!

Sign up