Question

Constant Connection Timeouts 110 Ubuntu

Posted March 12, 2021 315 views
MySQLUbuntu 18.04

I am experiencing numerous connection timeouts as of today.

I have one server running as live.rail-record.co.uk.
Live.rail-record.co.uk connects to member.rail-record.co.uk using fsockopen on port 3306 , to access some information from the mysql database.

Both websites are using Apache, and it is worth noting that I do NOT get timeouts when accessing member.rail-record.co.uk from browser.
My PHP settings for member.rail-record.co.uk - the site that is timing out on remote connections via SSH and fsockopen - can be found at https://member.rail-record.co.uk/php-info.php

I recently upgraded the server to a Intel 4gb 2cpu last weekend. The site was working fine up until today and nothing has since changed as far as I am aware.

  • When I login using PuTTY, I will get an error “network error: software caused connection abort, either after a few seconds regardless of whether the terminal is idle or not. I could be part way typing a command, it would freeze, and timeout.
  • I am using fsockopen on an external website, and occasionally this fails with Connection Timeout 110.
  • The SSH terminal runs sluggish.
  • When ping-ing the server from CMD on my local windows 10 machine, most pings work fine, but about 1 in 4 have 1 ping failure, reporting a timeout.

I have attempted to diagnose the problem in a number of ways.

  • I have "ufw allow” -ed my origin server IP which is connecting to the second server experiencing the timeouts.
  • I have disabled UFW altogether, ruling out UFW blocking my severs connections.
  • In my.cnf for mysql, I have increased the max packet size.

But even with the above, that shouldnt cause the sluggishness and timeouts when using the SSH?

Does any one have any recommendations as to what I can do about this?

Here’s some more information.

At 10:26, I triggered the timeout. But nothing in mysql error log for 10:26, only an earlier timeout:

2021-03-12T10:09:38.596997Z 420 [Note] Got timeout reading communication packets

There are a number of lines in UFW along the lines of the below (although not excessive, maybe a few per minute), but as I mentioned, disabling UFW has no effect and nothing about my live server IP / origin requesting server

Mar 12 10:27:37 test kernel: [ 1806.742533] [UFW BLOCK] IN=eth0 OUT= MAC=aa:70:04:4c:5a:d0:fe:00:00:00:01:01:08:00 SRC=195.54.161.151 DST=206.189.114.48 LEN=40 TOS=0x0$
Mar 12 10:27:38 test kernel: [ 1807.438358] [UFW BLOCK] IN=eth0 OUT= MAC=aa:70:04:4c:5a:d0:fe:00:00:00:01:01:08:00 SRC=195.54.161.151 DST=206.189.114.48 LEN=40 

Guessing no point checking apache logs as site accessible fine, just external connections.

I have a user set up on mysql for the remote site, and I am binding address etc in my.cnf. Like I said, everything fine until today.

Does anyone have any suggestions of what else to check, to help diagnose the issue?

Thanks

1 comment
  • Should have mentioned, the timing out server is on IP 206.189.114.48 and is proxied by cloudflare. The timeout happens regardless of whether cloudflare proxy is enabled or not.

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
Submit an Answer
1 answer

Hi there,

As far as I can see there has been a network maintenance today which could have possibly caused the timeouts that you’ve seen:

https://status.digitalocean.com/incidents/xjvkssr50lnc

If your Droplet is not in that region, another thing that I could suggest is checking your server resources utilization either via your DigitalOcean control panel and checking the monitoring graphs there or via SSH and using the htop command for example. It is possible that the connection is timing out due to a very high load on the server.

Let me know how it goes.

Regards,
Bobby

  • Thanks for your reply Bobby.

    The server is in LON1 region so not affected by the maintenance.

    Server resources are no problem at all, server load below 1.0 and plenty of unused ram.

    It’s worth noting that my website traffic remains unchanged compared to before, and the affected droplet was managing fine on the 5 dollar server , so very low load, only basic mysql queries.

    Is it possible that another user on my stack could be overloading the connections , or is this throttled?

    • Hi there,

      What I personally do in such cases is to check the server logs in order to verify that there was no suspicious activity going on at that time.

      Besides the system logs, you could try using this script here to summarize the access logs of your web server as well.

      Regarding another user causing such high network load, it is highly unlikely as this would need to be a huge DDoS attack which would affect a lot of other services as well. And in such cases, I believe that DigitalOcean takes immediate actions. But if the issue continues, I think that it might be worth checking with the support team to verify that this is not the case.

      Regards,
      Bobby