Apache causing massive load spike after reboot

Question

My server is running Ubuntu 18.04.5 LTS, with LAMP stack (PHP 7.4), and apache version 2.4.29 using Prefork MPM. My server has 16 CPUs and 29GB of RAM. This server hosts 258 websites.

Whenever I reboot my server, I notice (using top command) that apache is creating a crazy amount of processes. This is completely overloading the server. This ONLY happens when I reboot the server, or I manually stop apache (service apache2 stop) and then start apache (service apache2 start).

Normally this is the output of top (during normal hours):

https://drive.google.com/file/d/163-N2dq_ARKAICQEwnWA0SaLPLxtlOls/view?usp=sharing

After rebooting or stopping and starting apache manually, the load average climbs to over 126.4! It slowly (over about 10-15 minutes) goes back to normal. For the first 5 minutes after rebooting or stopping and starting apache, the load just steadily climbs until it gets to about 126-130.

Here is my MPM configuration (/etc/apache2/mods-enabled/mpm_prefork.conf):

<IfModule mpm_prefork_module> StartServers 5 MinSpareServers 5 MaxSpareServers 10 MaxRequestWorkers 150 MaxConnectionsPerChild 0 </IfModule>

Also KeepAlive is ON, MaxKeepAliveRequests is 100, and KeepAliveTimeout is 2.

What is very strange, is I made an exact copy of the server, and placed a handful of sites (6 websites) on the copy, and when I reboot, or start and stop apache on this copy server, this behavior does not happen. This is an EXACT copy of the server mentioned above, it is just only hosting 6 websites (WAY less traffic) instead of 258.

Could this problem be caused by too many client connections waiting on apache to respond during reboot, and when apache server finally comes back up it floods apache with connections?

Would increasing the values in my MPM configuration possibly help this? I’ve checked logs, and I never go over MaxRequestWorkers.

It has been suggested that I move from mpm_prefork to php-fpm and mod_event but for certain reasons this is not practical for me. I need to keep mpm_prefork.

alexdo · Answer

Hello, @alhxcex As you mentioned the issue is most probably caused by the fact that too many connections are waiting for the Apache server to respond. Also, the fact that you’re using mpm_prefork which does not support multi-threading is causing Apache to create multiple requests which overload the server in case of a reboot or restart of the Apache service. If you switch to mpm_worker_module or mpm_event_module you will experience a significant difference in the performance. Event is based on the worker MPM, which implements a hybrid multi-process multi-threaded server. A single control process (the parent) is responsible for launching child processes. Each child process creates a fixed number of server threads as specified in the ThreadsPerChild directive, as well as a listener thread that listens for connections and passes them to a worker thread for processing when they arrive. You can check the information here: https://httpd.apache.org/docs/current/mod/worker.html https://httpd.apache.org/docs/2.4/mod/event.html You can also play adjust the Apache workers configuration and try to tweak them and also change the php-fpm pool settings in order to optimize the settings based on the websites traffic. In my experience, with mpm_prefork_module the performance will always suffer due to the lack of multi-process multi-threading. Hope that this helps!Regards,Alex

KFSys · Answer

The issue is likely caused by a combination of factors related to the high number of hosted sites, their configurations, and client requests hitting the server when Apache restarts. Let’s analyze the situation and explore solutions while keeping your MPM prefork configuration. Root Cause Analysis High Number of Waiting Connections on Restart : When Apache restarts, it stops responding to incoming requests, leading to a backlog of queued client connections. When it comes back up, it tries to handle all these queued connections simultaneously, causing a spike in processes and load. MPM Prefork Configuration : Your StartServers value is likely too low for the number of sites and connections being served. When Apache starts, it rapidly spawns new processes to handle incoming requests until it reaches MaxRequestWorkers. This aggressive spawning can overload the server. KeepAlive Settings : With KeepAlive enabled and a KeepAliveTimeout of 2, Apache holds connections open longer, exacerbating resource usage during a flood of requests. Difference Between Original and Copy Server : The copy server handles fewer websites and traffic, so it doesn’t experience the same connection backlog during a restart. Recommended Actions 1. Adjust MPM Prefork Configuration Update your mpm_prefork.conf to better handle the server’s load during restarts: StartServers 20 MinSpareServers 20 MaxSpareServers 40 MaxRequestWorkers 300 MaxConnectionsPerChild 1000 Explanation : StartServers: Increase to 20 so Apache can handle initial load spikes more gracefully. MinSpareServers and MaxSpareServers: Provide more spare processes to handle queued requests after a restart. MaxRequestWorkers: Increase cautiously if you’re confident the server can handle more connections. Monitor this to avoid running out of memory. MaxConnectionsPerChild: Use a finite value (e.g., 1000) to periodically recycle processes and release memory. 2. Reduce KeepAliveTimeout Lowering the KeepAliveTimeout reduces the time Apache holds each connection open, freeing resources for new requests: KeepAliveTimeout 1 A lower timeout is especially useful during heavy traffic periods. 3. Implement Graceful Restarts Avoid using stop and start commands when restarting Apache. Instead, use: service apache2 reload or apachectl graceful This allows Apache to finish handling current connections before restarting, reducing the number of queued connections. 4. Analyze Log Files Enable detailed logging to identify problematic sites or spikes in traffic during restarts: Access Logs : Check /var/log/apache2/access.log for patterns of excessive or malicious connections. Error Logs : Look for errors or warnings in /var/log/apache2/error.log. 5. Rate-Limit Incoming Connections Use the Apache mod_ratelimit module or tools like fail2ban or iptables to limit the number of simultaneous connections per client during restarts: Example using mod_ratelimit: SetEnv rate-limit 100 ` 6. Consider DNS TTL for Hosted Sites If your hosted websites use DNS with a low TTL, browsers may aggressively re-request connections during Apache restarts. Increase TTL values to reduce this behavior. 7. Monitor Server Performance Use tools like htop, atop, or Apache mod_status to monitor real-time server performance and adjust configuration as needed.

Become a contributor for community

DigitalOcean Documentation

Resources for startups and AI-native businesses

The developer cloud

Start building today

Report this

Apache causing massive load spike after reboot

Related questions

DigitalOcean repository for MariaDB 10.3 is broken

How to upgrade the distribution used by my droplet?

I just used ssh to do-release-upgrade from ubuntu 18.04 to 20.04 and now site is down

Unable to Reset Root Password in Ubuntu 18.04 LTS

Need to install php.7.4 on Ubuntu 18. Can't get it in my list to update.

What is the best Linux (or at least ABI-compatible) OS distribution to use for building a custom image for small 1GB 1cpu VMs?

eth0 not getting configured correctly after upgrade.

Cannot upgrade mysql from 5.7 to 8 (Ubunto 18)

Hosting 3 Sites On the Same Droplet using subdomains

Permission Denied (Pubkey) whilst deploying Terragraph

Dashboard shows older version of Ubuntu after several upgrades.

Error during upgrade Ubuntu 18.04 to 22.04

Report this

Apache causing massive load spike after reboot

Related questions

DigitalOcean repository for MariaDB 10.3 is broken

How to upgrade the distribution used by my droplet?

I just used ssh to do-release-upgrade from ubuntu 18.04 to 20.04 and now site is down

Unable to Reset Root Password in Ubuntu 18.04 LTS

Need to install php.7.4 on Ubuntu 18. Can't get it in my list to update.

What is the best Linux (or at least ABI-compatible) OS distribution to use for building a custom image for small 1GB 1cpu VMs?

eth0 not getting configured correctly after upgrade.

Cannot upgrade mysql from 5.7 to 8 (Ubunto 18)

Hosting 3 Sites On the Same Droplet using subdomains

Permission Denied (Pubkey) whilst deploying Terragraph

Dashboard shows older version of Ubuntu after several upgrades.

Error during upgrade Ubuntu 18.04 to 22.04