Related

Can't install php-soap to my PHP 7.0 centos Question Question
cpu from 13% to 100% in one minute Question Question

Question

Why my CentOS droplet is eating all the RAM memory?

Posted February 4, 2020 1.4k views
CentOSServer OptimizationMonitoring

I started with a 2vCPU 2GB CentOS 7 x64 droplet installed from marketplace with a Plesk image. I have only create a Wordpress website with no more than 400 visits per month and followed the initial server setup for CentOS 7 on DigitalOcean.

Then I have started getting alerts from the DigitalOcean monitoring service because the droplet was using more than 70% of memory. I receive those mails 7 times per day, memory usage and resolved ones.

So I thought I was running out of memory, then I have purchased 2GB extra. So my droplet is now a 2vCPU 4GB. After 3 days, same as before, the alert emails for memory beyond 70% started to get my inbox.

At this particular moment, the server is upside that 70% of memory so I ran the command I know DigitalOcean uses to calculate memory (https://www.digitalocean.com/docs/monitoring/resources/glossary-of-terms/#memory)

cat /proc/meminfo

The results are:

MemTotal:   3880364 kB
MemFree:          287080 kB
MemAvailable:    2561780 kB
Buffers:              32 kB
Cached:           994716 kB
SwapCached:            0 kB
Active:          1224116 kB
Inactive:         455532 kB
Active(anon):     769364 kB
Inactive(anon):    27284 kB
Active(file):     454752 kB
Inactive(file):   428248 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                16 kB
Writeback:             0 kB
AnonPages:        684948 kB
Mapped:           101816 kB
Shmem:            111748 kB
Slab:            1750432 kB
SReclaimable:    1678876 kB
SUnreclaim:        71556 kB
KernelStack:        5552 kB
PageTables:        17372 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     1940180 kB
Committed_AS:    3978072 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       95708 kB
VmallocChunk:   34359537660 kB
HardwareCorrupted:     0 kB
AnonHugePages:    344064 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       98276 kB
DirectMap2M:     4096000 kB
DirectMap1G:           0 kB

The command free -m results are:

                     total          used        free      shared buff/cache  available
Mem:           3789         935         240         109        2613        2464
Swap:             0           0           0

Plesk on the other side, trough Grafana is telling me that
memory:memory-used:value 858 Mb
memory:memory-cached:value 971 Mb
memory:memory-slab_recl:value 1.6 Gb

Any ideas? Anything that would help me understand this situation and solve it?

Add a comment
danielpcharrua
By:
danielpcharrua

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
2 answers
nusbaum February 7, 2020

It doesn’t look like an overwhelming number of HTTP processes running, but check your access_log for lots of requests to XML-RPC. That seems to be a favorite attack point and enough requests can easily overwhelm a droplet. I would also check your security log, /var/log/secure on centos, for ssh login attempts. I recently had a server overwhelmed by login attempts and had to put a more aggressive fail2ban policy in place.

Reply Report
  • danielpcharrua February 8, 2020

    @nusbaum I had some requests to xmlrpc.php file you are right on that… I have blocked all requests with Apache. For the ssh login attempts, for the moment are only mine, I think this is because I have changed the default port…

    Thank you for the information provided!

    Reply Report
KFSys February 4, 2020

Hi @danielpcharrua,

I’ll recommend using the top or htop command. These commands will show you the top commands using CPU,RAM, time it has been running etc.

It’s possible the problem is related to a lot of traffic or the websites/applications are too heavy which generates the RAM being eaten. You’ll be able to confirm this with the above commands. Additionally, you can use the ps aux command to see all the processes and see which one you would need to either kill or further investigate why it’s taking so much memory.

Additionally, check if you haven’t actually configured any cron jobs which spawn and create traffic. The cron jobs might be generating MySQL queries which does tend to take a lot of Memory if they are being executed for more than 60 seconds.

It’s good to enable slow query log for MySQL to see if there are any slow queries. If there are, I’ll recommend working towards optimizing your queries.

Lastly, I believe it will be best to test this as soon as you receive an e-mail about high load.

Regards,
KDSys

Reply Report
  • danielpcharrua February 6, 2020

    Hello @KDSys, the issue related with lot of traffic is not possible, there is only one website installed on Plesk with 100 visits in the last month.

    The cron jobs activated are only the ones that come with Plesk and are maintenance ones.

    I have enabled the slow query log for MySQL, I’ll update here if there is some relevant entry on that file.

    An important note was that mysql proc was consuming 6-7% of memory on top command before activating the slow query, after activating and because was restarted, is 3.4%. Let’s see what happen in the next hours-days.

    I’ll also share the output of the first lines of the top command now, for you to see if there is anything that could require attention. Remember is a 2 vCPU 4Gb droplet, only with Plesk and one website. Now DigitalOcean monitoring indicates memory use at 50%.

    Thank you for your help.

    top - 08:06:38 up 4 days, 17:34,  1 user,  load average: 0,00, 0,04, 0,08
Tasks: 128 total,   1 running, 127 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0,8 us,  0,5 sy,  0,0 ni, 98,7 id,  0,0 wa,  0,0 hi,  0,0 si,  0,0 st
MiB Mem :   3789,4 total,    247,7 free,    786,7 used,   2754,9 buff/cache
MiB Swap:      0,0 total,      0,0 free,      0,0 used.   2595,3 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                             
14159 mysql     20   0 1431860 133356   9456 S   1,0  3,4   2:32.23 mysqld                                                                              
26117 apache    20   0 1615500 126588   3996 S   0,0  3,3   0:09.23 httpd                                                                               
27467 apache    20   0 1615520 125404   4060 S   0,3  3,2   0:07.19 httpd                                                                               
26118 apache    20   0 1615384 125284   4100 S   0,0  3,2   0:09.47 httpd                                                                               
26116 apache    20   0 1287632 124512   3984 S   0,0  3,2   0:09.03 httpd                                                                               
 1272 root      20   0  410980 121064   7748 S   0,0  3,1   0:45.71 httpd                                                                               
26114 apache    20   0  409804 114492   1164 S   0,0  3,0   0:00.00 httpd                                                                               
26115 apache    20   0  410980 114244    932 S   0,0  2,9   0:00.45 httpd                                                                               
 1047 named     20   0  239828  61384   2016 S   0,0  1,6   0:01.22 named                                                                               
13329 psaadm    20   0  510940  57960  42416 S   1,3  1,5   2:07.51 sw-engine-fpm                                                                       
 1780 grafana   20   0 1294288  46880  11992 S   0,0  1,2  11:45.95 grafana-server                                                                      
13331 psaadm    20   0  506792  46392  35132 S   0,0  1,2   2:02.49 sw-engine-fpm                                                                       
27949 psaadm    20   0  506032  39348  28656 S   0,0  1,0   0:31.29 sw-engine-fpm                                                                       
30426 psaadm    20   0  506028  34660  24192 S   0,0  0,9   0:06.81 sw-engine-fpm                                                                       
26715 psaadm    20   0  371112  34044  16576 S   0,0  0,9   0:00.31 sw-engine                                                                           
26718 psaadm    20   0  369372  33152  17164 S   0,0  0,9   0:50.17 sw-engine                                                                           
24305 psaadm    20   0  369064  30880  16692 S   0,0  0,8   0:01.68 sw-engine                                                                           
25799 root      20   0  502812  23724  18104 S   0,0  0,6   0:11.06 sw-engine-fpm                                                                       
 2544 root      20   0 1419480  23612   5076 S   0,3  0,6  38:42.87 fail2ban-server                                                                     
 1009 root      20   0  585212  22236  13644 S   0,0  0,6   0:50.60 php-fpm                                                                             
  999 root      20   0  574204  18448   5184 S   0,0  0,5   1:13.07 tuned                                                                               
 1463 do-agent  20   0  309616  17828   5336 S   0,0  0,5   1:50.51 do-agent                                                                            
 2063 grafana   20   0  390012  16792   4136 S   0,0  0,4   3:13.26 simple-json-plu                                                                     
  697 polkitd   20   0  716236  12012   4076 S   0,0  0,3   0:05.71 polkitd                                                                             
 1476 root      20   0  232804  11084   8020 S   0,0  0,3   0:36.89 rsyslogd                                                                            
  467 root      20   0   48424   9544   9116 S   0,0  0,2   0:12.18 systemd-journal                                                                     
26710 root      20   0  144892   6920   1664 S   0,0  0,2   0:00.11 plesk                                                                               
    1 root      20   0   46336   6620   4000 S   0,3  0,2   2:46.91 systemd                                                                             
30865 root      20   0  167336   6524   5064 S   0,0  0,2   0:00.39 sshd                                                                                
29403 sw-cp-s+  20   0   54424   5916   2396 S   0,0  0,2   0:21.88 sw-cp-serverd                                                                       
25648 root      20   0 1018204   5884   3412 S   0,0  0,2   2:11.44 sw-collectd                                                                         
29695 nginx     20   0   25596   4944   2636 S   0,0  0,1   0:14.73 nginx                                                                               
29402 root      20   0   52412   4788   3216 S   0,0  0,1   0:00.02 sw-cp-serverd                                                                       
22498 root      20   0   47304   4456   2872 S   0,0  0,1   0:00.06 systemd-udevd                                                                       
 1478 root      20   0  112920   4344   3316 S   0,0  0,1   0:00.07 sshd                                                                                
 4443 postfix   20   0   96856   4340   3300 S   0,0  0,1   0:18.61 qmgr
    Reply Report
    • KFSys February 9, 2020

      Hi @danielpcharrua,

      The provided top information shows you are using only 700 MBs out of 4GB. Having said that, I think I know what may have deceived you:

      MiB Mem :   3789,4 total,    247,7 free,    786,7 used,   2754,9 buff/cache

      If you check the second column, it says 247,7 free MB RAM however you actually have 2754,9 MB RAM cached and ready to use.

      Regards,
      KDSys

      Reply Report
      • danielpcharrua February 10, 2020

        Hi @KDSys, I fully understand that I have 2754,9 MB available.. Thank you!

        Reply Report
        • KFSys February 10, 2020

          Hi @danielpcharrua,

          Can you confirm if your issue appeared again after the changes you made blocking your xmlrpc.php files and enabling the slow-query log or everything seems to have normalized?

          Regards,
          KDSys

          Reply Report
          • danielpcharrua February 10, 2020

            Hi @KDSys,

            I’m not having this issue at this particular moment, but I wanted to let the server ran for a couple of days. I’ll post here any updates.

            The MySql slow query log has no entries, so this was not the problem I think… Regarding to the xmlrpc.php file, it was visited but not too much, now is blocked.

            Thank you so much for the follow up and help.

            Reply Report
Submit an Answer

Related

Looking for something else?

Ask a new question Search for more help