Everything spiked and got 5XX error and NGINX error - [alert] 12197#0: *2936829

Posted November 10, 2021 125 views
NginxApacheServer Optimization

Suddenly the CPU and Bandwidth spiked on its own 2 times and the site went down for 5-10 mins.

Image -
Example link

Checking logs saw this error - [alert] 12197#0: *2936829 socket() failed (24: Too many open files) while connecting to upstream on NGINX

100’s of these error entries from the same 4-5 IPS.

Example link
Image -

Read a few places, to increase files, some said it could be server-side problem contact admins. Can someone help with a solution so we do not face this.

website - www.gadgetbridge.

P.S strange thing was when I tried opening DO community page, I got a 502 error here also.

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

Submit an Answer
2 answers

Hi @SP1,

There are two ways to go about this.

The first would be to increase the limits on your Nginx configuration to see if that will appear again.

You can change the maximum number of file descriptors a process can create by modifying the /etc/sysctl.conf file and adding the fs.file-max setting. Set fs.file-max=50000 to allow processes to create 50000 file descriptors.

Then you can modify the security limits in /etc/security/limits.conf. You can set these by adding two lines nginx soft nofile 10000 and nginx hard nofile 30000 . Then run sysctl -p to verify our change.

Finally, open up your /etc/nginx/nginx/conf file and add the workerrlimitnofile directive like so workerrlimitnofile 30000;. Then reload nginx.

The second suggestion is that you’ve been a target of a DDoS and it’s not something specific with your Droplet to worry about. If it was me, I’ll wait before making the above changes, I don’t think they are necessary most of the time and see when the next time this error appears the IP addresses that are accessing your website.

  • Thanks, yes some forums did suggest that it was a DDOS, which is bad, don’t know why people do that and what fun they get out of it, we will wait for sometime before we put in the lines you suggested. Thanks.

    • Hi @SP1,

      Don’t take it personally, people just seem to have a lot of time on their hands or are just having fun making other’s people lives harders.

      You can prevent it by blocking the IPs that are accessing the website if of course, it doesn’t seem legit.

      Are you expecting traffic from USA? Additionally, if it’s only one IP does it make wayyyyy to much traffic then it’s supposed to? You can think about services like fail2ban which will help you prevent that or using Cloudflare as a CDN.

It happened again today, same IP address all from one point in Kansas.