Ubuntu 16.04 webserver consistent intermittent freezing
I’ve got a droplet running Ubuntu 16.04, kernel version 4.4.0-200. It has been running for a few years with regular system updates and occasional configuration adjustments. No adjustments have been made recently. It serves as a webserver for a moderate traffic site and has held up pretty well in the time it’s been running.
Right now we’re seeing that very consistently - every minute - the server will freeze for about 10 seconds. I’ve tried several things to attempt to identify the cause but I can’t find it. CPU load is intense enough that even process monitors halt, so when the freeze ends and I check the process log (output to a file) I can clearly see the last entry before the freeze and the first entry after, but nothing during the freeze. CPU usage hovers around 80% idle. Running mpstat recurring, I can sometimes see it drop to single-digit idle - though mpstat also freezes so typically I just see before/after cpu idle. System memory is largely free; I just rebooted the server so only about 10% of system memory is currently in use, though that will grow to around 60% as MySQL memory caches build up.
I’m at a loss as to the cause of the problem, or how to further diagnose. I’ve dug through system logs multiple times, paying particular attention to any entries around that freeze point but nothing looks relevant, certainly nothing repetitive stands out. There are some recurring cron jobs running every minute, but these are user-level php scripts that have been running about as long as the server has, and nothing I’m seeing points to any issues with their execution.
These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.×