Website suddenly went down, Ubuntu / Apache server is still up

I got an email alert from the monitoring service I use, UptimeRobot about my site being down. The site is https://vendbits.com

In the Chrome web browser, I see the error message:

This site can’t be reached vendbits.com took too long to respond.

I’m able to login to the server via SSH fine.

When I run service --status-all in terminal, I get:

[ - ] apache-htcacheclean [ + ] apache2 [ + ] apparmor [ + ] apport [ + ] atd [ - ] console-setup.sh [ + ] cron [ - ] cryptdisks [ - ] cryptdisks-early [ + ] dbus [ + ] fail2ban [ - ] grub-common [ - ] hwclock.sh [ - ] irqbalance [ - ] iscsid [ + ] kdump-tools [ + ] kexec [ + ] kexec-load [ - ] keyboard-setup.sh [ + ] kmod [ - ] lvm2 [ - ] lvm2-lvmpolld [ + ] multipath-tools [ + ] mysql [ - ] open-iscsi [ - ] open-vm-tools [ - ] plymouth [ - ] plymouth-log [ + ] postfix [ + ] procps [ - ] rsync [ + ] rsyslog [ - ] screen-cleanup [ + ] ssh [ + ] udev [ + ] ufw [ + ] unattended-upgrades [ - ] uuidd [ - ] x11-common

Which all seems normal, I suppose?

I updated and upgraded all packages using:

sudo apt update & sudo apt upgrade

I checked the logs in /var/log/syslog/ and the most recent entries are cron jobs executing:

Jul 2002:35:01 vendbits CRON[60039]: (root) CMD (wget -O - https://vendbits.com/XXX >/dev/null 2>&1) Jul 20 02:35:01 vendbits CRON[60041]: (root) CMD (wget -O - https://vendbits.com/XXX >/dev/null 2>&1) Jul 20 02:35:01 vendbits CRON[60040]: (root) CMD (wget -O - https://vendbits.com/XXX >/dev/null 2>&1) Jul 20 02:35:01 vendbits CRON[60042]: (root) CMD (wget -O - https://vendbits.com/XXX >/dev/null 2>&1) Jul 20 02:35:01 vendbits CRON[60043]: (root) CMD (wget -O - https://vendbits.com/XXX >/dev/null 2>&1) Jul 20 02:35:01 vendbits CRON[60047]: (root) CMD (wget -O - https://vendbits.com/admin/updateCryptoPrices >/dev/null 2>&1) Jul 20 02:39:01 vendbits CRON[60053]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi) Jul 20 02:39:22 vendbits systemd[1]: Starting Clean php session files... Jul 20 02:39:23 vendbits sessionclean[60092]: PHP Warning: Module "exif" is already loaded in Unknown on line 0 Jul 20 02:39:23 vendbits sessionclean[60092]: PHP Warning: Module "fileinfo" is already loaded in Unknown on line 0 Jul 20 02:39:23 vendbits sessionclean[60092]: PHP Warning: Module "gd" is already loaded in Unknown on line 0 Jul 20 02:39:23 vendbits sessionclean[60092]: PHP Warning: Module "imagick" is already loaded in Unknown on line 0 Jul 20 02:39:23 vendbits sessionclean[60092]: PHP Warning: Module "mbstring" is already loaded in Unknown on line 0 Jul 20 02:39:23 vendbits systemd[1]: phpsessionclean.service: Succeeded. Jul 20 02:39:23 vendbits systemd[1]: Finished Clean php session files. Jul 20 02:40:01 vendbits CRON[60150]: (root) CMD (wget -O - URL-REMOVED >/dev/null 2>&1) Jul 20 02:40:01 vendbits CRON[60151]: (root) CMD (wget -O - URL-REMOVED >/dev/null 2>&1) Jul 20 02:40:01 vendbits CRON[60152]: (root) CMD (wget -O - URL-REMOVED >/dev/null 2>&1) Jul 20 02:40:01 vendbits CRON[60153]: (root) CMD (wget -O - URL-REMOVED >/dev/null 2>&1) Jul 20 02:40:01 vendbits CRON[60154]: (root) CMD (wget -O - URL-REMOVED >/dev/null 2>&1) Jul 20 02:40:01 vendbits CRON[60160]: (root) CMD (wget -O - URL-REMOVED >/dev/null 2>&1) Jul 20 02:45:01 vendbits CRON[60203]: (root) CMD (wget -O - URL-REMOVED >/dev/null 2>&1) Jul 20 02:45:01 vendbits CRON[60202]: (root) CMD (wget -O - URL-REMOVED >/dev/null 2>&1) Jul 20 02:45:01 vendbits CRON[60204]: (root) CMD (wget -O - URL-REMOVED >/dev/null 2>&1) Jul 20 02:45:01 vendbits CRON[60205]: (root) CMD (wget -O - URL-REMOVED >/dev/null 2>&1) Jul 20 02:45:01 vendbits CRON[60206]: (root) CMD (wget -O - URL-REMOVED >/dev/null 2>&1) Jul 20 02:45:01 vendbits CRON[60210]: (root) CMD (wget -O - URL-REMOVED >/dev/null 2>&1) Jul 20 02:50:01 vendbits CRON[60559]: (root) CMD (wget -O - URL-REMOVED >/dev/null 2>&1) Jul 20 02:50:01 vendbits CRON[60560]: (root) CMD (wget -O - URL-REMOVED >/dev/null 2>&1) Jul 20 02:50:01 vendbits CRON[60561]: (root) CMD (wget -O - URL-REMOVED >/dev/null 2>&1) Jul 20 02:50:01 vendbits CRON[60562]: (root) CMD (wget -O - URL-REMOVED >/dev/null 2>&1) Jul 20 02:50:01 vendbits CRON[60563]: (root) CMD (wget -O - URL-REMOVED >/dev/null 2>&1) Jul 20 02:50:01 vendbits CRON[60567]: (root) CMD (wget -O - URL-REMOVED >/dev/null 2>&1)

I checked disk space using df

Filesystem 1K-blocks Used Available Use% Mounted on udev 899424 0 899424 0% /dev tmpfs 183388 992 182396 1% /run /dev/vda1 50620216 12721684 37882148 26% / tmpfs 916932 0 916932 0% /dev/shm tmpfs 5120 0 5120 0% /run/lock tmpfs 916932 0 916932 0% /sys/fs/cgroup /dev/loop0 69504 69504 0 100% /snap/lxd/22753 /dev/loop3 63488 63488 0 100% /snap/core20/1494 /dev/loop1 69632 69632 0 100% /snap/lxd/22526 /dev/loop2 56960 56960 0 100% /snap/core18/2538 /dev/loop4 56960 56960 0 100% /snap/core18/2409 /dev/loop6 48128 48128 0 100% /snap/snapd/16292 /dev/loop5 63488 63488 0 100% /snap/core20/1518 /dev/vda15 106858 5321 101537 5% /boot/efi /dev/loop7 48128 48128 0 100% /snap/snapd/16010 tmpfs 183384 0 183384 0% /run/user/0

All that seems normal.

I check apache error logs /var/log/apache2/error.log

[Wed Jul 20 00:00:03.106315 2022] [mpm_prefork:notice] [pid 1313657] AH00163: Apache/2.4.41 (Ubuntu) OpenSSL/1.1.1f configured -- resuming normal operations [Wed Jul 20 00:00:03.106349 2022] [core:notice] [pid 1313657] AH00094: Command line: '/usr/sbin/apache2' [Wed Jul 20 00:39:42.145217 2022] [mpm_prefork:notice] [pid 1313657] AH00169: caught SIGTERM, shutting down PHP Warning: Module "exif" is already loaded in Unknown on line 0 PHP Warning: Module "fileinfo" is already loaded in Unknown on line 0 PHP Warning: Module "gd" is already loaded in Unknown on line 0 PHP Warning: Module "imagick" is already loaded in Unknown on line 0 PHP Warning: Module "mbstring" is already loaded in Unknown on line 0 [Wed Jul 20 00:39:59.186682 2022] [mpm_prefork:notice] [pid 838] AH00163: Apache/2.4.41 (Ubuntu) OpenSSL/1.1.1f configured -- resuming normal operations [Wed Jul 20 00:39:59.186762 2022] [core:notice] [pid 838] AH00094: Command line: '/usr/sbin/apache2' [Wed Jul 20 01:11:10.282503 2022] [mpm_prefork:notice] [pid 838] AH00169: caught SIGTERM, shutting down PHP Warning: Module "exif" is already loaded in Unknown on line 0 PHP Warning: Module "fileinfo" is already loaded in Unknown on line 0 PHP Warning: Module "gd" is already loaded in Unknown on line 0 PHP Warning: Module "imagick" is already loaded in Unknown on line 0 PHP Warning: Module "mbstring" is already loaded in Unknown on line 0 [Wed Jul 20 01:11:26.156646 2022] [mpm_prefork:notice] [pid 837] AH00163: Apache/2.4.41 (Ubuntu) OpenSSL/1.1.1f configured -- resuming normal operations [Wed Jul 20 01:11:26.156734 2022] [core:notice] [pid 837] AH00094: Command line: '/usr/sbin/apache2' [Wed Jul 20 01:23:47.910850 2022] [mpm_prefork:notice] [pid 837] AH00169: caught SIGTERM, shutting down PHP Warning: Module "exif" is already loaded in Unknown on line 0 PHP Warning: Module "fileinfo" is already loaded in Unknown on line 0 PHP Warning: Module "gd" is already loaded in Unknown on line 0 PHP Warning: Module "imagick" is already loaded in Unknown on line 0 PHP Warning: Module "mbstring" is already loaded in Unknown on line 0 [Wed Jul 20 01:23:48.120158 2022] [mpm_prefork:notice] [pid 2329] AH00163: Apache/2.4.41 (Ubuntu) OpenSSL/1.1.1f configured -- resuming normal operations [Wed Jul 20 01:23:48.120259 2022] [core:notice] [pid 2329] AH00094: Command line: '/usr/sbin/apache2'

According to serverfault post

“these are not errors, they are just informational notices. error.log is misnamed; much of what goes to it is informational only. – ysth May 21, 2015 at 17:25”

/var/log/apache2/access.log latest entries are the same cron jobs as in /var/log/syslog/

137.184.142.4 - - [20/Jul/2022:03:10:02 +0000] "GET /XXX-URL-HIDDEN HTTP/1.1" 200 6384 "-" "Wget/1.20.3 (linux-gnu)" 137.184.142.4 - - [20/Jul/2022:03:10:02 +0000] "GET /XXX-URL-HIDDEN HTTP/1.1" 200 7424 "-" "Wget/1.20.3 (linux-gnu)" 137.184.142.4 - - [20/Jul/2022:03:10:02 +0000] "GET /XXX-URL-HIDDEN HTTP/1.1" 200 6449 "-" "Wget/1.20.3 (linux-gnu)" 137.184.142.4 - - [20/Jul/2022:03:15:01 +0000] "GET /URL-HIDDEN HTTP/1.1" 200 6384 "-" "Wget/1.20.3 (linux-gnu)" 137.184.142.4 - - [20/Jul/2022:03:15:01 +0000] "GET /URL-HIDDEN HTTP/1.1" 200 7424 "-" "Wget/1.20.3 (linux-gnu)" 137.184.142.4 - - [20/Jul/2022:03:15:01 +0000] "GET /URL-HIDDEN HTTP/1.1" 200 6384 "-" "Wget/1.20.3 (linux-gnu)" 137.184.142.4 - - [20/Jul/2022:03:15:01 +0000] "GET /URL-HIDDEN HTTP/1.1" 200 7427 "-" "Wget/1.20.3 (linux-gnu)" 137.184.142.4 - - [20/Jul/2022:03:15:01 +0000] "GET /URL-HIDDEN HTTP/1.1" 200 6384 "-" "Wget/1.20.3 (linux-gnu)" 137.184.142.4 - - [20/Jul/2022:03:15:01 +0000] "GET /URL-HIDDEN HTTP/1.1" 200 6449 "-" "Wget/1.20.3 (linux-gnu)" ::1 - - [20/Jul/2022:03:15:02 +0000] "OPTIONS * HTTP/1.0" 200 126 "-" "Apache/2.4.41 (Ubuntu) OpenSSL/1.1.1f (internal dummy connection)" 137.184.142.4 - - [20/Jul/2022:03:20:01 +0000] "GET /URL-HIDDEN HTTP/1.1" 200 7424 "-" "Wget/1.20.3 (linux-gnu)" 137.184.142.4 - - [20/Jul/2022:03:20:01 +0000] "GET /URL-HIDDEN HTTP/1.1" 200 6384 "-" "Wget/1.20.3 (linux-gnu)" 137.184.142.4 - - [20/Jul/2022:03:20:01 +0000] "GET /URL-HIDDEN HTTP/1.1" 200 6384 "-" "Wget/1.20.3 (linux-gnu)" 137.184.142.4 - - [20/Jul/2022:03:20:01 +0000] "GET /URL-HIDDEN HTTP/1.1" 200 7427 "-" "Wget/1.20.3 (linux-gnu)" 137.184.142.4 - - [20/Jul/2022:03:20:01 +0000] "GET /URL-HIDDEN HTTP/1.1" 200 6384 "-" "Wget/1.20.3 (linux-gnu)" 137.184.142.4 - - [20/Jul/2022:03:20:01 +0000] "GET /URL-HIDDEN HTTP/1.1" 200 6449 "-" "Wget/1.20.3 (linux-gnu)"

The domain registration was renewed last month.

The firewall was disabled when I checked using sudo ufw status

What else should I be checking?

The only other thing I could think of is somebody created a listing (vendbits is a marketplace for digital products) for a pirated version of FL studio, and I received a message through the privacyguardian email system 6 days ago for a listing removal request. The site only receives about 2 visitors a day since it’s new and I didn’t get around to removing the listing. Was the domain blacklisted through ICANN, blocked by DigitalOcean or something?

Thank you in advance.