Question

Connection timed out from Russia Only

Posted August 23, 2019 1k views
Nginx Ubuntu Node.js Load Balancing

Hi
In the last couple of month, I noticed that a few Russian users complained that my app isn’t working in Russia. This is very strange since all my other users can connect, I can connect, and Postman monitor doesn’t see any problems.

So to investigate this issue, I found a Russian version of Postman monitoring to do a serer check from Russia and surely enough - Connection timed out.

I went a step further and rented a virtual server in Russia and ran a few tests:
nslookup resolved the correct IP, however any GET / POST requests just times out:
curl: (7) Failed to connect to <domain> port 443: Connection timed out

Has anyone had something like this?
Thanks

Add a comment
deishelon
By:
deishelon
Add a comment

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

1 answer
bobbyiliev August 23, 2019

Hello,

What you could try doing is running a traceroute from your Russian server to your domain name. That way you would be able to see where exactly the connection is timing out.

Also if you are using a CDN like CloudFlare make sure that there are no Geo restrictions rules there.

Another thing that you could check is the firewall rules on your droplet itself.

Hope that this helps!
Regards,
Bobby

Reply Report
  • deishelon August 23, 2019

    traceroute directly to the server’s IP goes fine:

    traceroute to <IP> (<IP>), 30 hops max, 60 byte packets
 1  192.168.1.1 (192.168.1.1)  0.441 ms  0.491 ms  0.397 ms
 2  10.251.14.99 (10.251.14.99)  6.054 ms 10.251.14.97 (10.251.14.97)  6.073 ms 10.251.14.99 (10.251.14.99)  6.131 ms
 3  10.251.14.98 (10.251.14.98)  3.492 ms  3.483 ms 10.251.14.96 (10.251.14.96)  3.432 ms
 4  95.167.94.213 (95.167.94.213)  12.980 ms 95.167.37.213 (95.167.37.213)  13.197 ms 95.167.94.213 (95.167.94.213)  13.041 ms
 5  217.161.68.34 (217.161.68.34)  140.517 ms  144.559 ms  141.743 ms
 6  * * *
 7  ntt-gw-pcr1.fnt.cw.net (195.2.21.98)  140.292 ms  139.599 ms  149.333 ms
 8  ae-12.r25.frnkge08.de.bb.gin.ntt.net (129.250.4.186)  159.811 ms  150.871 ms  150.598 ms
 9  ae-9.r25.amstnl02.nl.bb.gin.ntt.net (129.250.3.77)  151.118 ms  153.918 ms  149.112 ms
10  ae-1.a01.amstnl02.nl.bb.gin.ntt.net (129.250.4.173)  146.803 ms  152.164 ms  156.488 ms
11  * * *
12  138.197.244.71 (138.197.244.71)  157.961 ms 138.197.244.81 (138.197.244.81)  150.815 ms 138.197.244.71 (138.197.244.71)  157.881 ms
13  * 138.197.251.237 (138.197.251.237)  152.525 ms *
14  <IP> (<IP>)  148.386 ms  153.137 ms  149.577 ms

    Ping is also ok

    PING <IP> (<IP>) 56(84) bytes of data.
64 bytes from <IP>: icmp_seq=1 ttl=58 time=145 ms
64 bytes from <IP>: icmp_seq=2 ttl=58 time=144 ms
64 bytes from <IP>: icmp_seq=3 ttl=58 time=144 ms
64 bytes from <IP>: icmp_seq=4 ttl=58 time=144 ms

    However, even directly to the server’s IP wget or curl can’t get any responses from the server. Where I can get from my local computer. I’m not using any CDN services.

    The firewall is disabled on the server at the moment:

    $ sudo ufw status
Status: inactive
    Reply Report
  • deishelon August 23, 2019

    After a bit more digging I found that my server receives the connection from the Russian server
    sudo netstat -tupn | grep <RUSSIAN IP>

    tcp 0 0 <IP>:80 <RUSSIAN IP> SYN_RECV -

    But it seems to stuck in SYN_RECV state

    Reply Report
    • bobbyiliev August 23, 2019

      Hello,

      In this case it is something on the server itself that is rejecting the connections. I would recommend checking if you have GeoIP enabled or possibly Fail2Ban.

      Also check if you have CSF installed.

      Let me know how it goes!
      Regards,
      Bobby

      Reply Report
      • deishelon August 23, 2019

        Fail2Ban and CSF are not installed.

        It looks like I do have some GeoIP packages installed

        dpkg -l | grep geo

        ii  geoip-database                 20181108-1                          all          IP lookup command line tools that use the GeoIP library (country database)
ii  libgeoip1:amd64                1.6.12-1                            amd64        non-DNS IP-to-country resolver library
ii  libnginx-mod-http-geoip        1.15.9-0ubuntu1.1                   amd64        GeoIP HTTP module for Nginx

        But Nginx.conf doesn’t use it. And anyway, the same things happens if I try to connect directly to Node js (same SYN_RECV)

        iptables configuration is also all to ACCEPT

        $ iptables -L

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

        Also, I have tried Creating a brand new droplet (Ubuntu 19.04 x64) - the same thing, I can connect, Russian connection - SYN_RECV

        Is there any other package, system config that I might need to have a look at?

        Reply Report
        • bobbyiliev August 23, 2019

          What I could suggest is trying to sniff the package with tcpdump:

          tcpdump -nnvvXSs 1514 -i any host your.russian.ip.here

          This should provide you with some information on what exactly is going on.

          I think that the best thing to do is to actually use strace to see what exactly the process is doing. So to do that use:

          lsof -i :80

          Find the PID of the process that was generated by your Russian server connection, then strace this process:

          strace -vvv -s 1000 -ff -p 28594

          This would give you an output of the exact things that the process is doing and you should be able to see where it is getting stuck.

          Hope that this helps!

          Reply Report
          • deishelon August 25, 2019

            Ok, I’ve tried this way and wasn’t getting any useful data from PID of the prosses.
            So I’ve created a new droplet on Debian (instead of Ubuntu), and tada! Russian server can connect with no problem. So obviously there is a problem with the Ubuntu distribution.

            But! Although Russian server can access the end server directly, I use Load Balancer and Russian server times out if I try to access via the load balancer. 😲

            TL; DL Russian server can access the end server directly, but times out via load balancer. I guess under the hood you guys use Ubuntu for load balancing?
            Is there any way to migrate my load balancer to Debian or let DO’s sysadmins to have a look?

            PS. My local computer can connect with no problems

            Reply Report
          • bobbyiliev August 26, 2019

            Hi @deishelon,

            Regarding the LB problem, I would recommend raising a ticket with DigitalOcean Support team and they will be able to advise you further on that.

            Regarding the Ubuntu issue, I’ll test that at my end and I’ll let you know how it goes!

            Regards,
            Bobby

            Reply Report
Submit an Answer

Related

Looking for something else?

Ask a new question Search for more help