Site just died out of the blue - not even sure how to debug what happened

June 3, 2016 4k views
Logging Apache CMS WordPress PHP DigitalOcean

I setup a 10$/mo droplet, running a WordPress install.

Things went smoothly, I encountered no issues and was able to get everything setup quite easily.

fast forward 23 days, and my site is now hitting a 504 - Gateway Time-Out error.

I reached out to the digital ocean staff (not expecting to get much help - which I didn’t) and was told to run ‘top’ to check what processes are running on the server.

Yesterday, around 7:00PM EST, I can see in the chart that’s when my CPU usage started jumping from about 15-22% to 100+%. I haven’t logged into the site, sshed in, ftped in, made any alterations etc.

I’m wondering why would my site just start spiking north of 100%, when I’m not receiving any more traffic than I normally am.

I’m at a loss here - and the sites been down for over 24 hours now, and I’ve made little to no progress in debugging it.

Thinking it might be time to call it quits and transfer away from DO - due to the down times.

Thanks to anyone who is willing to shead some light.

Graph Screenshot:

You can see, that in the past day is when everything started going haywire.

  • Did you run ‘top’ to see what processes are running on the server?

  • @gparent I did, here is the output:

    Lots of php-fpm instances running . CPU is around 80%, but the numbers don’t add up with what is listed in ‘top’.

  • 80 divided by 3.7 ~== 20 which seems to match the number of processes in your screenshot, or is at least close enough. I would check your web server’s access and error logs for the requests that those PHP-FPM processes are handling, and see if they are normal or if you’re under some sort of attack.

  • @gparent How do you block ALL incoming GET requests?

  • You could turn off your web server with service nginx stop

2 Answers

You probably got hacked. Transferring away from one unmanaged provider to the other does not fix it.

  • Highly doubt the site was hacked - additionally, I wouldn’t be going to another un-managed provider.

    • What makes you doubt it? How often did you run security updates on the server?

It looks like I have a tremendous number of incoming GET requests to a specific RSS feed URL that doesn’t exist.

All of the HTTP responses are 301 and 404. Is there any way to block ALL of these requests? I have a good feeling this is what’s taking down my server.

It seems that I am getting about 20-25 every 2-3 seconds. In the time I wrote this I received just about 1,705 incoming requests.

  • You would need to show us the URL in question for us to give any specific recommendation. Otherwise, I would use nginx’s location blocks to handle the request.

    • The URL in the request is ?feed=ads and returns either a 301 or a 404. At last check there were over 50,000 requests flooding my server.

      At this point I’m just going to look into hiring someone. I’m not a linux/apache guy - I’m a full stack dev, with no sysadmin skills - and my business is just loosing money. My site being down for 24hours+ is enough for me to just hire someone. Surprised at how easy things were to setup, but one minor hiccup and it’s all downhill.

      If it were something I did I’d be cool with it - but this all happened last night when I was out to dinner, which leads me to believe something changed at the server level or on my account.

Have another answer? Share your knowledge.