Hi guys,
Any idea why CPU load average spikes every 1-1.5 hour?
Here are some MySQL error logs for 1.5 hours (one cycle) as well as syslog for 1.5 hours (one cycle).
Thanks a lot!
BR, Tom
Subscribe
Share
Submit an answer
You can type!ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
Sign in to AnswerThese answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.
Hey @Azinity
your mysql instance is being killed regularly by Linux’s OOM (out of memory) killer. You should review the load consumed by the resources running on the server. Based on experience it’s usually down to a greedy query that causes a resource contention and makes Linux sacrifice the most expensive process.
At a high level, here’s my recommendations;
pt-query-digestYou might get some joy through upgrading the droplet to something larger but that may only buy you time.
BR
Andrew
Hi @bobbyiliev,
This is what we got in the jail.local file. Some folks said that it’s better to have both http-get-conf and http-post-conf files because GET may not catch the POST attacks. But yet this doesn’t help!
Hi @bobbyiliev,
Appreciate your responsiveness and suggestions.
We have installed and configured fail2ban and modsecurity but they don’t help fend off those botnets which are attacking us every hour.
I believe that we will have to use ufw to painstakingly deny blocks of IPs! The following resources are a good start to block botnets.
http://www.countryipblocks.net/ http://www.okean.com/thegoods.html
Any idea?
BR, Tom
Hi @bobbyiliev,
Thank you so much for your reply.
We have a firewall set up for ports 22, 80 and 443 but firewalls are not meant to mitigate large-scale DDoS attacks. Firewalls are overwhelmed and rendered useless. We got hundreds of requests sending from a five different IP addresses in just 10 seconds.
CDN is a good way to fend off DDoS attacks but it is website-specific. However, if you have 30+ websites to manage, would it be better to protect the server as a whole than websites individually?
As an expert in system administration and a long-time contributor in the DO community, can you tell if DO droplets are protected against DDoS attacks? I read some comments from some DO folks DO droplets are protected against DDoS attacks so why are our websites being constantly attacked then?
How can we tell from apache access logs which website they are attacking at a particular point in time?
Your invaluable suggestions would be highly appreciated.
Thank you so much in advance.
BR, Tom
Hi @bobbyiliev,
Thanks for your responsiveness. But how do you currently protect your droplet(s) from DDoS attacks assuming you are using DO? Cloudflare?
In fact, some of the websites have Wordfence installed so it should be able to mitigate DDoS attacks. However, some sites don’t. I guess those are the vulnerable sites which have caused trouble.
We installed mod-evasive last night but the spikes keep coming back since then! We have set it up correctly.
We still need to figure out why it doesn’t work. This is definitely a DDoS attack, isn’t this?
BR, Tom
Hi @bobbyiliev,
Thank you for your suggestion.
This is a DDoS attack, isn’t this? So I can simply install an Apache module, ModEvasive to fend off any future DDoS attack instead.
BR, Tom
hi @bobbyiliev,
Below are two lists of malicious IPs using your bash script taken from two different time frames when the CPU load spiked. The IPs are different every time. Any ideas will be highly appreciated.
BR, Tom
995 222.79.50.74 China 837 110.167.93.145 China 772 124.235.138.14 China 722 223.166.74.9 China 682 113.128.105.94 China 628 27.211.56.183 China 556 1.30.28.77 China 549 222.94.212.104 China 543 58.244.10.241 China 445 222.94.195.46 China 432 121.57.12.85 China 361 121.57.229.55 China 294 113.128.105.226 China
987 121.57.224.247 China 906 106.45.1.160 China 855 113.128.104.139 China 727 113.200.71.104 China 711 36.5.180.204 China 694 60.208.210.52 China 629 113.57.114.56 China 622 222.94.140.87 China 597 222.74.205.247 China 526 219.143.174.114 China 421 220.175.61.238 China 407 123.139.42.19 China 401 123.179.7.121 China 350 58.19.92.17 China 338 36.47.163.62 China 302 123.138.77.50 China 277 1.80.145.196 China
hi @bobbyiliev,
Thank you so much for your suggestion.
Today I have come to realize that the access log I pasted earlier was an hour behind because the time the DO graphs indicates in our local computer time which UTC+1 and the server’s time is UTC. No wonder I didn’t find anything. I have found many malicious IP addresses as follows which originate from China. However, they vary from time to time so it will literally be futile to block these IPs. So far we blocked ten IP but they keep changing! What do you suggest in this case?
Thank you so much.
BR, Tom
hi @bobbyiliev,
one quick question to you regarding your bash script.
Most Recent top 20 IP addresses:
how recent are they? the last 10 days? thanks a lot!
besides, no ip address from china shows up on this list. can we arrive at the conclusion that for the last 10 days during which mysqld was killed, it wasn’t the so-called unidentified bots from china which crippled our sites?
Most Recent top 20 IP addresses: 185 84.202.100.14 Norway 104 139.59.96.33 Singapore 87 62.210.143.10 France 73 183.89.212.199 Thailand 69 36.237.55.63 Taiwan 59 180.215.255.141 India 31 110.235.33.135 India 16 77.75.77.101 Czech Republic 13 216.244.66.226 United States 10 34.73.85.242 United States 7 77.88.5.176 Russian Federation 6 199.58.86.211 United States 5 35.231.80.6 United States 5 216.244.66.230 United States 5 193.106.30.99 Ukraine 4 66.249.65.134 United States 4 64.202.185.246 United States 4 62.84.58.177 Kazakhstan 4 46.229.168.163 United States
br, tom
hi @bobbyiliev,
thank you so much for your suggestion.
if you follow our correspondence above, cron job is, in fact, the first thing we looked into. we have no cron job, and we never set up one.
we ran
crontab -lto list all scheduled cron jobs for the root user which is the only user, and we got “no crontab for root”someone mentioned it could be wp-cron.php but i doubt. we had 31 websites two months ago and we had no such issue at all. the issue started up to surface two weeks ago with the same sites. could it be a plugin which is causing that?
i posted the same on many forums and so far no one can crack it! i would be grateful if you could share more of your invaluable experience with us!!
thank you so much.
br, tom