Scale up as you grow — whether you're running one virtual machine or ten thousand.

From GPU-powered inference and Kubernetes to managed databases and storage, get everything you need to build, scale, and deploy intelligent applications.

100% reproducible:
Create Droplet -> Ubuntu 20.04
As root:
# journalctl -xe
-- No entries --
Well THAT’S CONCERNING. Let’s see if it’s actually there:
root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# ls /var/log/journal/* -al
total 8204
drwxr-sr-x+ 2 root systemd-journal 4096 Feb 18 07:02 .
drwxr-sr-x+ 3 root systemd-journal 4096 Feb 18 07:02 ..
-rw-r-----+ 1 root systemd-journal 8388608 Feb 18 07:05 system.journal
So, the system.journal file is there, but it isn’t being recognized by journalctl? Weird.
Let’s try turning it off and back on again…
root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# systemctl restart systemd-journald
root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# ls /var/log/journal/* -al
/var/log/journal/5392a32a818ae1e9b6e285a07776c843:
total 8204
drwxr-sr-x+ 2 root systemd-journal 4096 Feb 18 07:02 .
drwxr-sr-x+ 4 root systemd-journal 4096 Feb 18 07:06 ..
-rw-r-----+ 1 root systemd-journal 8388608 Feb 18 07:06 system.journal
/var/log/journal/ae60451991a4be968a3d976e620f44ad:
total 8200
drwxr-sr-x+ 2 root systemd-journal 4096 Feb 18 07:06 .
drwxr-sr-x+ 4 root systemd-journal 4096 Feb 18 07:06 ..
-rw-r-----+ 1 root systemd-journal 8388608 Feb 18 07:06 system.journal
So that just created another log file. Interesting. Let’s check systemd-journald:
root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# systemctl status systemd-journald
● systemd-journald.service - Journal Service
Loaded: loaded (/lib/systemd/system/systemd-journald.service; static; vendor preset: enabled)
Active: active (running) since Fri 2022-02-18 07:02:25 UTC; 3min 12s ago
TriggeredBy: ● systemd-journald.socket
● systemd-journald-dev-log.socket
● systemd-journald-audit.socket
Docs: man:systemd-journald.service(8)
man:journald.conf(5)
Main PID: 338 (systemd-journal)
Status: "Processing requests..."
Tasks: 1 (limit: 2339)
Memory: 3.9M
CGroup: /system.slice/systemd-journald.service
└─338 /lib/systemd/systemd-journald
root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# ls /var/log/journal/
5392a32a818ae1e9b6e285a07776c843
root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# ls /var/log/journal/*
system.journal
root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# ls /var/log/journal/* -al
total 8204
drwxr-sr-x+ 2 root systemd-journal 4096 Feb 18 07:02 .
drwxr-sr-x+ 3 root systemd-journal 4096 Feb 18 07:02 ..
-rw-r-----+ 1 root systemd-journal 8388608 Feb 18 07:05 system.journal
root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# systemctl restart systemd-journald
root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# ls /var/log/journal/* -al
/var/log/journal/5392a32a818ae1e9b6e285a07776c843:
total 8204
drwxr-sr-x+ 2 root systemd-journal 4096 Feb 18 07:02 .
drwxr-sr-x+ 4 root systemd-journal 4096 Feb 18 07:06 ..
-rw-r-----+ 1 root systemd-journal 8388608 Feb 18 07:06 system.journal
/var/log/journal/ae60451991a4be968a3d976e620f44ad:
total 8200
drwxr-sr-x+ 2 root systemd-journal 4096 Feb 18 07:06 .
drwxr-sr-x+ 4 root systemd-journal 4096 Feb 18 07:06 ..
-rw-r-----+ 1 root systemd-journal 8388608 Feb 18 07:06 system.journal
So it’s running, and I can see the new file being created. Let’s take a look to see if we are seeing logs now…
root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# journalctl -xe
-- Logs begin at Fri 2022-02-18 07:06:46 UTC, end at Fri 2022-02-18 07:06:46 UTC. --
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd[1]: Stopping Journal Service...
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd-journald[338]: Received SIGTERM from PID 1 (systemd).
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd[1]: systemd-journald.service: Succeeded.
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd[1]: Stopped Journal Service.
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd[1]: Starting Journal Service...
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd-journald[2041]: Journal started
-- Subject: The journal has been started
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The system journal process has started up, opened the journal
-- files for writing and is now ready to process requests.
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd-journald[2041]: System Journal (/var/log/journal/ae60451991a4be968a3d976e620f44ad) is 8.0M, max 4.0G, 3.9G free.
-- Subject: Disk space used by the journal
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- System Journal (/var/log/journal/ae60451991a4be968a3d976e620f44ad) is currently using 8.0M.
-- Maximum allowed usage is set to 4.0G.
-- Leaving at least 2.4G free (of currently available 46.7G of disk space).
-- Enforced usage limit is thus 4.0G, of which 3.9G are still available.
--
-- The limits controlling how much disk space is used by the journal may
-- be configured with SystemMaxUse=, SystemKeepFree=, SystemMaxFileSize=,
-- RuntimeMaxUse=, RuntimeKeepFree=, RuntimeMaxFileSize= settings in
-- /etc/systemd/journald.conf. See journald.conf(5) for details.
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd[1]: Started Journal Service.
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd[1]: Starting Flush Journal to Persistent Storage...
-- Subject: A start job for unit systemd-journal-flush.service has begun execution
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit systemd-journal-flush.service has begun execution.
--
-- The job identifier is 1023.
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd-journald[2041]: System Journal (/var/log/journal/ae60451991a4be968a3d976e620f44ad) is 8.0M, max 4.0G, 3.9G free.
-- Subject: Disk space used by the journal
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- System Journal (/var/log/journal/ae60451991a4be968a3d976e620f44ad) is currently using 8.0M.
-- Maximum allowed usage is set to 4.0G.
-- Leaving at least 2.4G free (of currently available 46.7G of disk space).
-- Enforced usage limit is thus 4.0G, of which 3.9G are still available.
--
-- The limits controlling how much disk space is used by the journal may
-- be configured with SystemMaxUse=, SystemKeepFree=, SystemMaxFileSize=,
-- RuntimeMaxUse=, RuntimeKeepFree=, RuntimeMaxFileSize= settings in
-- /etc/systemd/journald.conf. See journald.conf(5) for details.
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd[1]: Finished Flush Journal to Persistent Storage.
-- Subject: A start job for unit systemd-journal-flush.service has finished successfully
So a simple restart of the journal fixed it. Is this an issue on Ubuntu’s side? DigitalOcean’s side? I don’t know. All I know is that I now know nothing about my VMs until I go in there and restart the journal, which was unexpected behavior. (affecting both 20.04 and 18.04)
jhonnydepp12
dolphinnn
Vaishnavi Vundyala
016512ea60864da0867e5280f293de
Peter Monadjemi
Prince7hakur
devjk1
fc9acbfbc3124cbf8df28a1c5c5cdb
ab871885551143798f41908c709271
gouskova
Saivion Services