GlusterFS Geo-Replication

September 6, 2015 7.5k views
Clustering Ubuntu
hollar
By:
hollar

USEFUL INFO:

I have three nodes: nginx01-03 (all running Ubuntu 14.04). The nodes nginx01/02 currently reside in NYC3 and have GlusterFS (v3.6.5) replication working between them without issue via the private network. I have modified the /etc/hosts file of all three nodes to point to the respective parties IP address that's necessary (in this case: 01 and 02 see the private IP of each other and the public IP of 03 whereas 03 sees the public IP of 01 and 02). They've been given the shortname of nginx01-03 respectively.

ISSUE:

I was able to establish replication between two nodes in the same DC (NYC3) over the private network without issue; however I'm trying to utilize the geo-replication feature of GlusterFS with my NYC3 node(s) and a LON node.

I've attempted to use instructions found here: http://www.jamescoyle.net/how-to/1037-synchronise-a-glusterfs-volume-to-a-remote-site-using-geo-replication

However there are numerous issues that I've encountered:

  1. the secret.pem file did not allow for password-less SSH. I had to modify the ~/.ssh/config file on the master node to specifically point to that file by using the IdentityFile option.

  2. the command to initiate the volume on the master node to the slave node failed:

root@nginx01:~# gluster volume geo-replication primaryvol nginx03:/gluster/ start
Staging failed on localhost. Please check the log file for more details.
geo-replication command failed

I have no idea where the "log file" in question is; however I utilized the --log-file= option to specify one. It proved less than helpful as the only information I received was the following two lines:

root@nginx01:~# tail /var/log/gluster_error
[2015-09-06 12:51:34.949782] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now
[2015-09-06 12:51:35.205223] I [input.c:36:cli_batch] 0-: Exiting with: -1

Anyone have success setting this up or using geo-replication in general? Any additional information you need just let me know and I'll gladly provide!

1 comment
  • rgb000 September 28, 2015

    Hello. I succeed to setup geo-replication like you want on Ubuntu.
    Let say, you have one LAN volume nginx01-02 with name 'lan-volume' - this is Master volume.
    Slave volume is nginx3 with name 'geo-volume'.
    Now your settings should match next conditions:

    1. Size of Master volume should be less then Slave
    2. There should be passwordless ssh between all three nodes
    3. You should fix gsyncd bug on Ubuntu

    Condition 1. 

    root@client1:~# df -h
Filesystem                       Size  Used Avail Use% Mounted on
...
nginx01_02:/geo-volume          158G  5.0G  145G   4% /mnt/geo_volume
nginx3:/lan-volume              79G  2.5G   73G   4% /mnt/lan_volume
...

    Total size of Master(lan-volume) is less then Slave(geo-volume).

    Condition 2.
    Make passwordless ssh for root. Generate authorized and private key: 

    root@nginx01:~# ssh-keygen -t rsa -f /root/.ssh/id_rsa -q -N ""
root@nginx01:~# mv /root/.ssh/id_rsa.pub /root/.ssh/authorized_keys
root@nginx01:~# ls /root/.ssh/
authorized_keys  id_rsa

    Now sync folder to all nodes.

    root@nginx01:~# rsync -avP /root/.ssh/ nginx02:/root/.ssh/
root@nginx01:~# rsync -avP /root/.ssh/ nginx03:/root/.ssh/

    Now connect to each of them to add to known hosts and check connection. 

    root@nginx01:~# ssh nginx02
The authenticity of host 'nginx02 (192.168.1.103)' can't be established.
ECDSA key fingerprint is 11:5d:55:29:8a:77:d8:08:b4:00:9b:a3:61:93:fe:e5.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'nginx01' (ECDSA) to the list of known hosts.
root@nginx02:~# 

root@nginx01:~# ssh nginx03
The authenticity of host 'nginx02 (192.168.1.103)' can't be established.
ECDSA key fingerprint is 31:yd:75:89:85:54:h6:65:b4:00:9b:a3:55:45:e5:23.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'nginx03' (ECDSA) to the list of known hosts.
root@nginx03:~#

    Condition 3.
    Geo-replication is using 'gsyncd' and it should be located in /usr/libexec/glusterfs/. On Ubuntu this pass doesn't exist. So you need to create it and make symbolic link to existing one all three nodes

    root@nginx01:~# mkdir -p /usr/libexec/glusterfs/     
root@nginx01:~# ln -s /usr/lib/x86_64-linux-gnu/glusterfs/gsyncd /usr/libexec/glusterfs/gsyncd 
root@nginx02:~# mkdir -p /usr/libexec/glusterfs/     
root@nginx02:~# ln -s /usr/lib/x86_64-linux-gnu/glusterfs/gsyncd /usr/libexec/glusterfs/gsyncd 
root@nginx03:~# mkdir -p /usr/libexec/glusterfs/     
root@nginx03:~# ln -s /usr/lib/x86_64-linux-gnu/glusterfs/gsyncd /usr/libexec/glusterfs/gsyncd

    Now you just need to login to Master, setup geo-volume and start geo-replication as in instruction you give link to.

Log In to Comment
1 Answer
asb MOD June 7, 2016

This question was answered by @rgb000:

Hello. I succeed to setup geo-replication like you want on Ubuntu.
Let say, you have one LAN volume nginx01-02 with name 'lan-volume' - this is Master volume.
Slave volume is nginx3 with name 'geo-volume'.
Now your settings should match next conditions:

  1. Size of Master volume should be less then Slave
  2. There should be passwordless ssh between all three nodes
  3. You should fix gsyncd bug on Ubuntu

Condition 1. 

root@client1:~# df -h
Filesystem                       Size  Used Avail Use% Mounted on
...
nginx01_02:/geo-volume          158G  5.0G  145G   4% /mnt/geo_volume
nginx3:/lan-volume              79G  2.5G   73G   4% /mnt/lan_volume
...

Total size of Master(lan-volume) is less then Slave(geo-volume).

Condition 2.
Make passwordless ssh for root. Generate authorized and private key: 

root@nginx01:~# ssh-keygen -t rsa -f /root/.ssh/id_rsa -q -N ""
root@nginx01:~# mv /root/.ssh/id_rsa.pub /root/.ssh/authorized_keys
root@nginx01:~# ls /root/.ssh/
authorized_keys  id_rsa

Now sync folder to all nodes.

root@nginx01:~# rsync -avP /root/.ssh/ nginx02:/root/.ssh/
root@nginx01:~# rsync -avP /root/.ssh/ nginx03:/root/.ssh/

Now connect to each of them to add to known hosts and check connection. 

root@nginx01:~# ssh nginx02
The authenticity of host 'nginx02 (192.168.1.103)' can't be established.
ECDSA key fingerprint is 11:5d:55:29:8a:77:d8:08:b4:00:9b:a3:61:93:fe:e5.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'nginx01' (ECDSA) to the list of known hosts.
root@nginx02:~# 

root@nginx01:~# ssh nginx03
The authenticity of host 'nginx02 (192.168.1.103)' can't be established.
ECDSA key fingerprint is 31:yd:75:89:85:54:h6:65:b4:00:9b:a3:55:45:e5:23.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'nginx03' (ECDSA) to the list of known hosts.
root@nginx03:~#

Condition 3.
Geo-replication is using 'gsyncd' and it should be located in /usr/libexec/glusterfs/. On Ubuntu this pass doesn't exist. So you need to create it and make symbolic link to existing one all three nodes

root@nginx01:~# mkdir -p /usr/libexec/glusterfs/     
root@nginx01:~# ln -s /usr/lib/x86_64-linux-gnu/glusterfs/gsyncd /usr/libexec/glusterfs/gsyncd 
root@nginx02:~# mkdir -p /usr/libexec/glusterfs/     
root@nginx02:~# ln -s /usr/lib/x86_64-linux-gnu/glusterfs/gsyncd /usr/libexec/glusterfs/gsyncd 
root@nginx03:~# mkdir -p /usr/libexec/glusterfs/     
root@nginx03:~# ln -s /usr/lib/x86_64-linux-gnu/glusterfs/gsyncd /usr/libexec/glusterfs/gsyncd

Now you just need to login to Master, setup geo-volume and start geo-replication as in instruction you give link to.

View the original comment

Reply
Have another answer? Share your knowledge.

Related Questions