Connection timed out from Russia Only

August 23, 2019 345 views
Load Balancing Nginx Node.js Ubuntu

Hi
In the last couple of month, I noticed that a few Russian users complained that my app isn’t working in Russia. This is very strange since all my other users can connect, I can connect, and Postman monitor doesn’t see any problems.

So to investigate this issue, I found a Russian version of Postman monitoring to do a serer check from Russia and surely enough - Connection timed out.

I went a step further and rented a virtual server in Russia and ran a few tests:
nslookup resolved the correct IP, however any GET / POST requests just times out:
curl: (7) Failed to connect to <domain> port 443: Connection timed out

Has anyone had something like this?
Thanks

1 Answer

Hello,

What you could try doing is running a traceroute from your Russian server to your domain name. That way you would be able to see where exactly the connection is timing out.

Also if you are using a CDN like CloudFlare make sure that there are no Geo restrictions rules there.

Another thing that you could check is the firewall rules on your droplet itself.

Hope that this helps!
Regards,
Bobby

  • traceroute directly to the server’s IP goes fine:

    traceroute to <IP> (<IP>), 30 hops max, 60 byte packets
     1  192.168.1.1 (192.168.1.1)  0.441 ms  0.491 ms  0.397 ms
     2  10.251.14.99 (10.251.14.99)  6.054 ms 10.251.14.97 (10.251.14.97)  6.073 ms 10.251.14.99 (10.251.14.99)  6.131 ms
     3  10.251.14.98 (10.251.14.98)  3.492 ms  3.483 ms 10.251.14.96 (10.251.14.96)  3.432 ms
     4  95.167.94.213 (95.167.94.213)  12.980 ms 95.167.37.213 (95.167.37.213)  13.197 ms 95.167.94.213 (95.167.94.213)  13.041 ms
     5  217.161.68.34 (217.161.68.34)  140.517 ms  144.559 ms  141.743 ms
     6  * * *
     7  ntt-gw-pcr1.fnt.cw.net (195.2.21.98)  140.292 ms  139.599 ms  149.333 ms
     8  ae-12.r25.frnkge08.de.bb.gin.ntt.net (129.250.4.186)  159.811 ms  150.871 ms  150.598 ms
     9  ae-9.r25.amstnl02.nl.bb.gin.ntt.net (129.250.3.77)  151.118 ms  153.918 ms  149.112 ms
    10  ae-1.a01.amstnl02.nl.bb.gin.ntt.net (129.250.4.173)  146.803 ms  152.164 ms  156.488 ms
    11  * * *
    12  138.197.244.71 (138.197.244.71)  157.961 ms 138.197.244.81 (138.197.244.81)  150.815 ms 138.197.244.71 (138.197.244.71)  157.881 ms
    13  * 138.197.251.237 (138.197.251.237)  152.525 ms *
    14  <IP> (<IP>)  148.386 ms  153.137 ms  149.577 ms
    

    Ping is also ok

    PING <IP> (<IP>) 56(84) bytes of data.
    64 bytes from <IP>: icmp_seq=1 ttl=58 time=145 ms
    64 bytes from <IP>: icmp_seq=2 ttl=58 time=144 ms
    64 bytes from <IP>: icmp_seq=3 ttl=58 time=144 ms
    64 bytes from <IP>: icmp_seq=4 ttl=58 time=144 ms
    

    However, even directly to the server’s IP wget or curl can’t get any responses from the server. Where I can get from my local computer. I’m not using any CDN services.

    The firewall is disabled on the server at the moment:

    $ sudo ufw status
    Status: inactive
    
  • After a bit more digging I found that my server receives the connection from the Russian server
    sudo netstat -tupn | grep <RUSSIAN IP>

    tcp 0 0 <IP>:80 <RUSSIAN IP> SYN_RECV -

    But it seems to stuck in SYN_RECV state

    • Hello,

      In this case it is something on the server itself that is rejecting the connections. I would recommend checking if you have GeoIP enabled or possibly Fail2Ban.

      Also check if you have CSF installed.

      Let me know how it goes!
      Regards,
      Bobby

      • Fail2Ban and CSF are not installed.

        It looks like I do have some GeoIP packages installed

        dpkg -l | grep geo

        ii  geoip-database                 20181108-1                          all          IP lookup command line tools that use the GeoIP library (country database)
        ii  libgeoip1:amd64                1.6.12-1                            amd64        non-DNS IP-to-country resolver library
        ii  libnginx-mod-http-geoip        1.15.9-0ubuntu1.1                   amd64        GeoIP HTTP module for Nginx
        
        

        But Nginx.conf doesn’t use it. And anyway, the same things happens if I try to connect directly to Node js (same SYN_RECV)

        iptables configuration is also all to ACCEPT

        $ iptables -L
        
        Chain INPUT (policy ACCEPT)
        target     prot opt source               destination         
        
        Chain FORWARD (policy ACCEPT)
        target     prot opt source               destination         
        
        Chain OUTPUT (policy ACCEPT)
        target     prot opt source               destination        
        
        

        Also, I have tried Creating a brand new droplet (Ubuntu 19.04 x64) - the same thing, I can connect, Russian connection - SYN_RECV

        Is there any other package, system config that I might need to have a look at?

        • What I could suggest is trying to sniff the package with tcpdump:

          tcpdump -nnvvXSs 1514 -i any host your.russian.ip.here
          

          This should provide you with some information on what exactly is going on.

          I think that the best thing to do is to actually use strace to see what exactly the process is doing. So to do that use:

          lsof -i :80
          

          Find the PID of the process that was generated by your Russian server connection, then strace this process:

          strace -vvv -s 1000 -ff -p 28594
          

          This would give you an output of the exact things that the process is doing and you should be able to see where it is getting stuck.

          Hope that this helps!

          • Ok, I’ve tried this way and wasn’t getting any useful data from PID of the prosses.
            So I’ve created a new droplet on Debian (instead of Ubuntu), and tada! Russian server can connect with no problem. So obviously there is a problem with the Ubuntu distribution.

            But! Although Russian server can access the end server directly, I use Load Balancer and Russian server times out if I try to access via the load balancer. 😲

            TL; DL Russian server can access the end server directly, but times out via load balancer. I guess under the hood you guys use Ubuntu for load balancing?
            Is there any way to migrate my load balancer to Debian or let DO’s sysadmins to have a look?

            PS. My local computer can connect with no problems

          • Hi @deishelon,

            Regarding the LB problem, I would recommend raising a ticket with DigitalOcean Support team and they will be able to advise you further on that.

            Regarding the Ubuntu issue, I’ll test that at my end and I’ll let you know how it goes!

            Regards,
            Bobby

Have another answer? Share your knowledge.