USEFUL INFO:
I have three nodes: nginx01-03 (all running Ubuntu 14.04). The nodes nginx01/02 currently reside in NYC3 and have GlusterFS (v3.6.5) replication working between them without issue via the private network. I have modified the /etc/hosts file of all three nodes to point to the respective parties IP address that’s necessary (in this case: 01 and 02 see the private IP of each other and the public IP of 03 whereas 03 sees the public IP of 01 and 02). They’ve been given the shortname of nginx01-03 respectively.
ISSUE:
I was able to establish replication between two nodes in the same DC (NYC3) over the private network without issue; however I’m trying to utilize the geo-replication feature of GlusterFS with my NYC3 node(s) and a LON node.
I’ve attempted to use instructions found here: http://www.jamescoyle.net/how-to/1037-synchronise-a-glusterfs-volume-to-a-remote-site-using-geo-replication
However there are numerous issues that I’ve encountered:
-
the
secret.pemfile did not allow for password-less SSH. I had to modify the~/.ssh/configfile on the master node to specifically point to that file by using theIdentityFileoption. -
the command to initiate the volume on the master node to the slave node failed:
root@nginx01:~# gluster volume geo-replication primaryvol nginx03:/gluster/ start
Staging failed on localhost. Please check the log file for more details.
geo-replication command failed
I have no idea where the “log file” in question is; however I utilized the --log-file= option to specify one. It proved less than helpful as the only information I received was the following two lines:
root@nginx01:~# tail /var/log/gluster_error
[2015-09-06 12:51:34.949782] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now
[2015-09-06 12:51:35.205223] I [input.c:36:cli_batch] 0-: Exiting with: -1
Anyone have success setting this up or using geo-replication in general? Any additional information you need just let me know and I’ll gladly provide!
These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.
This question was answered by @rgb000:
Hello. I succeed to setup geo-replication like you want on Ubuntu. Let say, you have one LAN volume nginx01-02 with name ‘lan-volume’ - this is Master volume. Slave volume is nginx3 with name ‘geo-volume’. Now your settings should match next conditions:
- Size of Master volume should be less then Slave
- There should be passwordless ssh between all three nodes
- You should fix gsyncd bug on Ubuntu
Condition 1.
root@client1:~# df -h Filesystem Size Used Avail Use% Mounted on ... nginx01_02:/geo-volume 158G 5.0G 145G 4% /mnt/geo_volume nginx3:/lan-volume 79G 2.5G 73G 4% /mnt/lan_volume ...Total size of Master(lan-volume) is less then Slave(geo-volume).
Condition 2. Make passwordless ssh for root. Generate authorized and private key:
root@nginx01:~# ssh-keygen -t rsa -f /root/.ssh/id_rsa -q -N "" root@nginx01:~# mv /root/.ssh/id_rsa.pub /root/.ssh/authorized_keys root@nginx01:~# ls /root/.ssh/ authorized_keys id_rsaNow sync folder to all nodes.
root@nginx01:~# rsync -avP /root/.ssh/ nginx02:/root/.ssh/ root@nginx01:~# rsync -avP /root/.ssh/ nginx03:/root/.ssh/Now connect to each of them to add to known hosts and check connection.
root@nginx01:~# ssh nginx02 The authenticity of host 'nginx02 (192.168.1.103)' can't be established. ECDSA key fingerprint is 11:5d:55:29:8a:77:d8:08:b4:00:9b:a3:61:93:fe:e5. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'nginx01' (ECDSA) to the list of known hosts. root@nginx02:~# root@nginx01:~# ssh nginx03 The authenticity of host 'nginx02 (192.168.1.103)' can't be established. ECDSA key fingerprint is 31:yd:75:89:85:54:h6:65:b4:00:9b:a3:55:45:e5:23. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'nginx03' (ECDSA) to the list of known hosts. root@nginx03:~#Condition 3. Geo-replication is using ‘gsyncd’ and it should be located in /usr/libexec/glusterfs/. On Ubuntu this pass doesn’t exist. So you need to create it and make symbolic link to existing one all three nodes
root@nginx01:~# mkdir -p /usr/libexec/glusterfs/ root@nginx01:~# ln -s /usr/lib/x86_64-linux-gnu/glusterfs/gsyncd /usr/libexec/glusterfs/gsyncd root@nginx02:~# mkdir -p /usr/libexec/glusterfs/ root@nginx02:~# ln -s /usr/lib/x86_64-linux-gnu/glusterfs/gsyncd /usr/libexec/glusterfs/gsyncd root@nginx03:~# mkdir -p /usr/libexec/glusterfs/ root@nginx03:~# ln -s /usr/lib/x86_64-linux-gnu/glusterfs/gsyncd /usr/libexec/glusterfs/gsyncdNow you just need to login to Master, setup geo-volume and start geo-replication as in instruction you give link to.
Hello. I succeed to setup geo-replication like you want on Ubuntu. Let say, you have one LAN volume nginx01-02 with name ‘lan-volume’ - this is Master volume. Slave volume is nginx3 with name ‘geo-volume’. Now your settings should match next conditions:
Condition 1.
Total size of Master(lan-volume) is less then Slave(geo-volume).
Condition 2. Make passwordless ssh for root. Generate authorized and private key:
Now sync folder to all nodes.
Now connect to each of them to add to known hosts and check connection.
Condition 3. Geo-replication is using ‘gsyncd’ and it should be located in /usr/libexec/glusterfs/. On Ubuntu this pass doesn’t exist. So you need to create it and make symbolic link to existing one all three nodes
Now you just need to login to Master, setup geo-volume and start geo-replication as in instruction you give link to.