Apache causing massive load spike after reboot

Hello, @alhxcex

As you mentioned the issue is most probably caused by the fact that too many connections are waiting for the Apache server to respond. Also, the fact that you’re using mpm_prefork which does not support multi-threading is causing Apache to create multiple requests which overload the server in case of a reboot or restart of the Apache service.

If you switch to mpm_worker_module or mpm_event_module you will experience a significant difference in the performance.

Event is based on the worker MPM, which implements a hybrid multi-process multi-threaded server. A single control process (the parent) is responsible for launching child processes. Each child process creates a fixed number of server threads as specified in the ThreadsPerChild directive, as well as a listener thread that listens for connections and passes them to a worker thread for processing when they arrive.

You can check the information here:

https://httpd.apache.org/docs/current/mod/worker.html

https://httpd.apache.org/docs/2.4/mod/event.html

You can also play adjust the Apache workers configuration and try to tweak them and also change the php-fpm pool settings in order to optimize the settings based on the websites traffic.

In my experience, with mpm_prefork_module the performance will always suffer due to the lack of multi-process multi-threading.

Hope that this helps! Regards, Alex

The issue is likely caused by a combination of factors related to the high number of hosted sites, their configurations, and client requests hitting the server when Apache restarts. Let’s analyze the situation and explore solutions while keeping your MPM prefork configuration.

Root Cause Analysis

1. High Number of Waiting Connections on Restart:

   • When Apache restarts, it stops responding to incoming requests, leading to a backlog of queued client connections. When it comes back up, it tries to handle all these queued connections simultaneously, causing a spike in processes and load.

2. MPM Prefork Configuration:

   • Your StartServers value is likely too low for the number of sites and connections being served. When Apache starts, it rapidly spawns new processes to handle incoming requests until it reaches MaxRequestWorkers. This aggressive spawning can overload the server.

3. KeepAlive Settings:

   • With KeepAlive enabled and a KeepAliveTimeout of 2, Apache holds connections open longer, exacerbating resource usage during a flood of requests.

4. Difference Between Original and Copy Server:

   • The copy server handles fewer websites and traffic, so it doesn’t experience the same connection backlog during a restart.

---

---

Recommended Actions

1. Adjust MPM Prefork Configuration

Update your mpm_prefork.conf to better handle the server’s load during restarts:

<IfModule mpm_prefork_module>
    StartServers          20
    MinSpareServers       20
    MaxSpareServers       40
    MaxRequestWorkers     300
    MaxConnectionsPerChild 1000
</IfModule>

Explanation:

• StartServers: Increase to 20 so Apache can handle initial load spikes more gracefully.
• MinSpareServers and MaxSpareServers: Provide more spare processes to handle queued requests after a restart.
• MaxRequestWorkers: Increase cautiously if you’re confident the server can handle more connections. Monitor this to avoid running out of memory.
• MaxConnectionsPerChild: Use a finite value (e.g., 1000) to periodically recycle processes and release memory.

---

2. Reduce KeepAliveTimeout

Lowering the KeepAliveTimeout reduces the time Apache holds each connection open, freeing resources for new requests:

KeepAliveTimeout 1

• A lower timeout is especially useful during heavy traffic periods.

---

3. Implement Graceful Restarts

Avoid using stop and start commands when restarting Apache. Instead, use:

service apache2 reload

or

apachectl graceful

• This allows Apache to finish handling current connections before restarting, reducing the number of queued connections.

---

4. Analyze Log Files

Enable detailed logging to identify problematic sites or spikes in traffic during restarts:

• Access Logs:
  • Check /var/log/apache2/access.log for patterns of excessive or malicious connections.
• Error Logs:
  • Look for errors or warnings in /var/log/apache2/error.log.

---

5. Rate-Limit Incoming Connections

Use the Apache mod_ratelimit module or tools like fail2ban or iptables to limit the number of simultaneous connections per client during restarts:

Example using mod_ratelimit:

<IfModule mod_ratelimit.c>     
    SetEnv rate-limit 100 
</IfModule>`

---

6. Consider DNS TTL for Hosted Sites

If your hosted websites use DNS with a low TTL, browsers may aggressively re-request connections during Apache restarts. Increase TTL values to reduce this behavior.

---

7. Monitor Server Performance

Use tools like htop, atop, or Apache mod_status to monitor real-time server performance and adjust configuration as needed.