Question

Why CPU load average spikes every 1-1.5 hour?

Hi guys,

Any idea why CPU load average spikes every 1-1.5 hour?

Here are some MySQL error logs for 1.5 hours (one cycle) as well as syslog for 1.5 hours (one cycle).

Thanks a lot!

BR, Tom

Logs: https://pastebin.com/UeRSVBmi


Submit an answer

This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

Sign In or Sign Up to Answer

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

Want to learn more? Join the DigitalOcean Community!

Join our DigitalOcean community of over a million developers for free! Get help and share knowledge in Q&A, subscribe to topics of interest, and get courses and tools that will help you grow as a developer and scale your project or business.

Hey @Azinity

your mysql instance is being killed regularly by Linux’s OOM (out of memory) killer. You should review the load consumed by the resources running on the server. Based on experience it’s usually down to a greedy query that causes a resource contention and makes Linux sacrifice the most expensive process.

At a high level, here’s my recommendations;

  1. Tune applications to be more conservative with resources. e.g. limit the amount of connections your web server can serve at once.
  • a. Turn on MySQL slow log and configure it to capture any query that takes longer than 0.5 seconds.
  • b. Analyse the slow query log using pt-query-digest
  • c. Tune the queries using indexing or rewrites to reduce the amount of resources the SQL queries need to use.

You might get some joy through upgrading the droplet to something larger but that may only buy you time.

BR

Andrew

Hi @bobbyiliev,

This is what we got in the jail.local file. Some folks said that it’s better to have both http-get-conf and http-post-conf files because GET may not catch the POST attacks. But yet this doesn’t help!

##Block the remote host that is trying to request suspicious URLs.
[apache-overflows]
enabled = true
port = http,https
filter = apache-overflows
logpath = /var/log/apache2/*error.log
maxretry = 4
bantime = 3600
ignoreip = 127.0.0.1/8 ::1 192.168.0.0/16

##Block the remote host that is trying to search for scripts on the website to execute.
[apache-noscript]
enabled = true
port = http,https
filter = apache-noscript
logpath = /var/log/apache2/*error.log
maxretry = 4
bantime = 3600
ignoreip = 127.0.0.1/8 ::1 192.168.0.0/16

##Block the remote host that is trying to request malicious bot.
[apache-badbots]
enabled = true
port = http,https
filter = apache-badbots
logpath = /var/log/apache2/*error.log
maxretry = 4
bantime = 3600
ignoreip = 127.0.0.1/8 ::1 192.168.0.0/16

##Simple attempt to block very basic DOS attacks over GET
##Tolerate ~3.3 GET/s in 30s (100 GET in less than 30s)
[http-get-dos]
enabled = true
port = http,https
filter = http-get-dos
logpath = /var/log/apache*/access.log
maxretry = 100
findtime = 30
bantime = 6000
ignoreip = 127.0.0.1/8 ::1 192.168.0.0/16
action = ufw[name=HTTP, port=http, protocol=tcp]

##Simple attempt to block very basic DOS attacks over POST
##Tolerate ~2 POST/s in 30s (60 POST in less than 30s)
[http-post-dos]
enabled = true
port = http,https
filter = http-post-dos
logpath   = /var/log/apache*/access.log
maxRetry = 60
findtime = 29
bantime = 6000
ignoreip = 127.0.0.1/8 192.168.0.0/16
action = iptables[name=HTTP, port=http, protocol=tcp]

##Block the failed login attempts on the SSH server.
[ssh]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/auth.log
maxretry = 4
bantime = 3600
ignoreip = 127.0.0.1/8 192.168.0.0/16

##Block attempts to use certain PHP behvior for malicious purpose
[php-url-fopen]
enabled = true
port    = http,https
filter  = php-url-fopen
logpath = /var/log/apache*/access.log

Hi @bobbyiliev,

Appreciate your responsiveness and suggestions.

We have installed and configured fail2ban and modsecurity but they don’t help fend off those botnets which are attacking us every hour.

I believe that we will have to use ufw to painstakingly deny blocks of IPs! The following resources are a good start to block botnets.

http://www.countryipblocks.net/ http://www.okean.com/thegoods.html

Any idea?

BR, Tom