Apache Keeps Crashing

January 3, 2017 345 views
Apache CentOS

I run a single Wordpress site on my CentOS 7 droplet, using Apache 2.4.6.

Every couple of days, my site suddenly becomes unavailable. I have to restart Apache, and it works again.

No errors, just I get a notification from my monitoring service that my site is down, and when I try to load it, it just hangs. No errors or anything.

And not a high server load, so I don't think it's just trying to load the site for a very long time or anything.

But, no errors that I can find. I've checked /etc/httpd/logs/errorlog, /etc/httpd/logs/sslerror_log, and even /var/log/messages. Also checked MariaDB just to be sure.

There are no errors. The only thing Apache says is when I restarted it after the fact.

I'm not sure if it could be due to memory, but I've checked the memory usage right around when it happens, and haven't noticed anything out of the ordinary.

I'm just not sure how to track this down. Any help would be appreciated.

5 Answers

Experiencing the same issues.... Seeking expert assistance

Without some more information it can be hard to find a cause.

One cause I see a lot is Apache can get stuck if the memory limit is hit on all instances open and timeout is disabled or made really long. If ignore user exit is turned on in Wordpress or PHP it will keep running a script even if in a loop and the user leaves.

Is there any CPU spikes around the same time? You might want to install a server monitor of some kind like New Relic that will give you info on what happens when it happens.

@programmer2188

To get a better idea of how you're setup:

1). What size is your Droplet? (512MB, 1GB, 2GB, etc)

2). Is Apache, PHP, and MariaDB the only software running?

3). Since you're using WordPress, what plugins, if any, do you have installed?
3a). Are they all up-to-date?

4). When running top via SSH, what's your CPU/RAM usage during:

  • Idle or Low-Traffic times; and
  • Peak or High-Traffic times
  • Hi,

    It's not been happening for a few days, but here are the answers to your questions:

    1. Droplet is 2 gb

    2. Yes that is the only software.

    3. Quite a few plugins, but all up-to-date:

    • Akismet
    • Auto Affiliate Links
    • bbPress
    • bbPress Genesis Extend
    • BBSpoiler
    • Better Click To Tweet
    • DW PromoBar
    • Easy Testimonials
    • EWWW Image Optimizer Cloud
    • Featured Image In Rss Feed
    • Genesis Layout Extras
    • Google Analytics Dashboard for WP
    • iframe
    • Jetpack by WordPress.com
    • Term Management Tools
    • ThirstyAffiliates
    • VaultPress
    • W3 Total Cache
    • WP ConvertKit
    • WP Customer Area
    • WP Customer Area - Conversations
    • WP Customer Area - Notifications
    • WP-Mail-SMTP
    • Yoast SEO Premium

    I know that's a lot, but I require all of them for various purposes.

    1. Right now's a pretty active time, so here's the top output, just the banner without the processes:
    top - 16:40:36 up 14 days, 20:09,  1 user,  load average: 0.32, 0.37, 0.35
    Tasks: 100 total,   1 running,  99 sleeping,   0 stopped,   0 zombie
    %Cpu(s):  2.8 us,  0.4 sy,  0.0 ni, 96.4 id,  0.1 wa,  0.0 hi,  0.0 si,  0.3 st
    KiB Mem :  1883728 total,   117324 free,   650200 used,  1116204 buff/cache
    KiB Swap:        0 total,        0 free,        0 used.   866944 avail Mem 
    

    The CPU's gone down a bit so here's another take from top:

    top - 16:49:01 up 14 days, 20:18,  1 user,  load average: 0.03, 0.15, 0.25
    Tasks:  98 total,   1 running,  97 sleeping,   0 stopped,   0 zombie
    %Cpu(s):  0.5 us,  0.0 sy,  0.0 ni, 99.2 id,  0.0 wa,  0.0 hi,  0.0 si,  0.3 st
    KiB Mem :  1883728 total,   210636 free,   556408 used,  1116684 buff/cache
    KiB Swap:        0 total,        0 free,        0 used.   961028 avail Mem 
    

    Like I said, Apache's been fine for about 5 or 6 days I'd say, but I didn't do anything to fix it, so I'm nervous about it happening again.

@programmer2188

Honestly, that's not as many plugins as I was expecting to be honest, so the count isn't an issue. I've seen 1-2GB instances run 2-3x that without any issues.

WordPress Plugins Overall

That said, one plugin that is always a concern in my eyes is EWWW Image Optimizer Cloud. Why? It has the potential to be a resource hog, especially if you're uploading quite a few images, or if you're allowing others to. For a small WordPress instance, it's really a non-issue, but if you upload a series of images and watch top, you can see what I mean. Because of the software it requires from the web server, and the overall intensiveness of said software, it can shoot CPU usage up to the ceiling and with it, your RAM (and lack of both means processes start hanging, fighting for resources, and ultimately crash when they can't get what they need).

While I understand the utility of the plugin, you're far better offer simply running them on your own prior to uploading them. Services such as the below are far better options:

https://compressor.io/
https://tinypng.com/

and on the commercial end:

https://kraken.io/

WordPress Plugins: W3 Total Cache

What are you setup to use? Are you using OpCode Caching, Memcached, Redis, File, or something else?

Optimizing MySQL

Login to SSH and grab your current MySQL configuration from /etc/mysql/my.cnf. While it normally takes a pretty busy site to bog dog even the most basic MySQL configuration, there's always one or two things we can tweak to speed things up -- or test.

Drop Apache, Use NGINX

Unless you're familiar with NGINX or the CLI, this is more of a last-ditch move. NGINX is ultimately more performant as is PHP-FPM over the default PHP module used by Apache. Using W3TC with NGINX, however, does require a little more love than it does with Apache, but it's really worth it.

What may cause high loads on Apache often results in seemingly reduced and low loads with NGINX.

  • Thanks for your answer.

    I don't think Ewww is the issue, as the only time it should be running is when I post a new blog post. But I'll definitely look into those services you recommended and see about dropping Ewww.

    Yeah I'm not really familiar with Nginx. I've been purely using Apache for years. Maybe I'll look into it though. What is the difficulty between W3TC and Nginx?

    Someone else was recommending I might use something like LiteSpeed with LSCache, or some other server-based caching solution, instead of W3TC. I have been having issues with W3TC caching the comment form and displaying one user's name and email to other users.

    Oh, for W3TC, I'm using opcode.

    I'll show my my.cnf, but there's really not much to see. I've not configured it at all yet.

    [mysqld]
    datadir=/var/lib/mysql
    socket=/var/lib/mysql/mysql.sock
    # Disabling symbolic-links is recommended to prevent assorted security risks
    symbolic-links=0
    # Settings user and group are ignored when systemd is used.
    # If you need to run mysqld under a different user or group,
    # customize your systemd unit file for mariadb according to the
    # instructions in http://fedoraproject.org/wiki/Systemd
    
    [mysqld_safe]
    log-error=/var/log/mariadb/mariadb.log
    pid-file=/var/run/mariadb/mariadb.pid
    
    #
    # include all files from the config directory
    #
    !includedir /etc/my.cnf.d
    
    

    There's also not much interesting in /etc/my.cnf.d/. Case in point, here's /etc/my.cnf.d/server.cnf.

    #
    # These groups are read by MariaDB server.
    # Use it for options that only the server (but not clients) should see
    #
    # See the examples of server my.cnf files in /usr/share/mysql/
    #
    
    # this is read by the standalone daemon and embedded servers
    [server]
    
    # this is only for the mysqld standalone daemon
    [mysqld]
    
    # this is only for embedded server
    [embedded]
    
    # This group is only read by MariaDB-5.5 servers.
    # If you use the same .cnf file for MariaDB of different versions,
    # use this group for options that older servers don't understand
    [mysqld-5.5]
    
    # These two groups are only read by MariaDB servers, not by MySQL.
    # If you use the same .cnf file for MySQL and MariaDB,
    # you can put MariaDB-only options here
    [mariadb]
    
    [mariadb-5.5]
    
    

    Thanks so much for the help.

@programmer2188

W3TC will generate a file when it detects NGINX instead of Apache and you'll essentially copy/paste the contents of that file in to your NGINX server block, then restart NGINX. The extra part I mentioned is because you have to manually do this anytime you make changes that would extend or remove any portion of the server block code that W3TC generates.

W3TC will generate changes made to your .htaccess file on the fly as long as it has the permissions to write to it. NGINX doesn't work that way since all of your configuration goes inside the server block for your domain.

That being said, NGINX + PHP-FPM, in my experience, is more performant than Apache + mod_php or similar. It can be used as a standard HTTP server, reverse proxy, caching proxy, etc. Also, should the need arise, you can actually separate NGINX from PHP-FPM and run them on their own servers. You would then configure NGINX to proxy the request to the PHP-FPM server. I actually just setup testing for that exact configuration.

MySQL Configuration

First off, before modifying your my.cnf file, make a backup copy:

cp /etc/mysql/my.cnf /usr/local/src/my.cnf

Verify the above backup is a valid copy and then shutdown MySQL:

service mysql stop

Then simply delete the existing file, create a new one, and then paste the contents below in.

sudo rm -rf /etc/mysql/my.cnf \
&& nano /etc/mysql/my.cnf
[mysql]
max_allowed_packet = 32M

[mysqld]
local-infile=0
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
symbolic-links=0

back_log = 75
max_connections = 300
key_buffer_size = 32M
myisam_sort_buffer_size = 32M
myisam_max_sort_file_size = 2048M
join_buffer_size = 64K
read_buffer_size = 64K
sort_buffer_size = 128K
table_definition_cache = 4096
table_open_cache = 2048
thread_cache_size = 64
wait_timeout = 1800
connect_timeout = 10
tmp_table_size = 32M
max_heap_table_size = 32M
max_allowed_packet = 32M
max_seeks_for_key = 1000
group_concat_max_len = 1024
max_length_for_sort_data = 1024
net_buffer_length = 16384
max_connect_errors = 100000
concurrent_insert = 2
read_rnd_buffer_size = 256K
bulk_insert_buffer_size = 8M
query_cache_limit = 512K
query_cache_size = 16M
query_cache_type = 1
query_cache_min_res_unit = 2K
query_prealloc_size = 262144
query_alloc_block_size = 65536
transaction_alloc_block_size = 8192
transaction_prealloc_size = 4096
default-storage-engine = InnoDB

log_warnings=1
slow_query_log=0
long_query_time=1
slow_query_log_file=/var/lib/mysql/slowq.log
log-error=/var/log/mysqld.log

innodb_large_prefix=1
innodb_purge_threads=1
innodb_doublewrite = 1

innodb_file_per_table = 1
innodb_open_files = 1000
innodb_data_file_path= ibdata1:10M:autoextend
innodb_buffer_pool_size = 48M
innodb_additional_mem_pool_size = 32M

innodb_log_files_in_group = 2
innodb_log_file_size = 64M
innodb_log_buffer_size = 8M
innodb_flush_log_at_trx_commit = 2
innodb_thread_concurrency = 0
innodb_lock_wait_timeout=50
#innodb_flush_method = O_DIRECT
innodb_support_xa=1

# 200 * # DISKS
innodb_io_capacity = 100
innodb_read_io_threads = 2
innodb_write_io_threads = 2

[mysqld_safe]
log-error=/var/log/mariadb/mariadb.log
pid-file=/var/run/mariadb/mariadb.pid

[mysqldump]
quick
max_allowed_packet = 32M

[myisamchk] 
key_buffer = 32M 
sort_buffer = 16M
read_buffer = 16M
write_buffer = 16M

[mysqlhotcopy]
interactive-timeout

Now start MySQL:

service mysql start

This is a bare minimum setup that should perform well for the time being. It enables query caching and provides tweaks to common configuration.

You shouldn't run in to any issues with that setup as long as you're running an up to date version of MySQL or MariaDB (a drop-in replacement for MySQL). If you do, that's why we created a backup so that we can revert back without issue.

If you do hit a snag, make sure to copy the error and paste it in a reply.

  • That configuration didn't work for some reason.

    [root@web1 html]# systemctl restart mariadb
    Job for mariadb.service failed because the control process exited with error code. See "systemctl status mariadb.service" and "journalctl -xe" for details.
    [root@web1 html]# systemctl status -l mariadb
    ● mariadb.service - MariaDB database server
       Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
       Active: failed (Result: exit-code) since Wed 2017-01-11 00:35:57 UTC; 5s ago
      Process: 25711 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID (code=exited, status=1/FAILURE)
      Process: 25710 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, status=0/SUCCESS)
      Process: 25682 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n (code=exited, status=0/SUCCESS)
     Main PID: 25710 (code=exited, status=0/SUCCESS)
    
    Jan 11 00:35:56 web1.cocreationcoaching.org systemd[1]: Starting MariaDB database server...
    Jan 11 00:35:57 web1.cocreationcoaching.org mysqld_safe[25710]: 170111 00:35:57 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.
    Jan 11 00:35:57 web1.cocreationcoaching.org mysqld_safe[25710]: 170111 00:35:57 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
    Jan 11 00:35:57 web1.cocreationcoaching.org mysqld_safe[25710]: 170111 00:35:57 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
    Jan 11 00:35:57 web1.cocreationcoaching.org systemd[1]: mariadb.service: control process exited, code=exited status=1
    Jan 11 00:35:57 web1.cocreationcoaching.org systemd[1]: Failed to start MariaDB database server.
    Jan 11 00:35:57 web1.cocreationcoaching.org systemd[1]: Unit mariadb.service entered failed state.
    Jan 11 00:35:57 web1.cocreationcoaching.org systemd[1]: mariadb.service failed.
    
    

    Obviously I restored the previous configuration. Should I also run the journalctl -xe command or is the above enough information?

Have another answer? Share your knowledge.