100% reproducible:
Create Droplet -> Ubuntu 20.04
As root
:
# journalctl -xe
-- No entries --
Well THAT’S CONCERNING. Let’s see if it’s actually there:
root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# ls /var/log/journal/* -al
total 8204
drwxr-sr-x+ 2 root systemd-journal 4096 Feb 18 07:02 .
drwxr-sr-x+ 3 root systemd-journal 4096 Feb 18 07:02 ..
-rw-r-----+ 1 root systemd-journal 8388608 Feb 18 07:05 system.journal
So, the system.journal file is there, but it isn’t being recognized by journalctl? Weird.
Let’s try turning it off and back on again…
root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# systemctl restart systemd-journald
root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# ls /var/log/journal/* -al
/var/log/journal/5392a32a818ae1e9b6e285a07776c843:
total 8204
drwxr-sr-x+ 2 root systemd-journal 4096 Feb 18 07:02 .
drwxr-sr-x+ 4 root systemd-journal 4096 Feb 18 07:06 ..
-rw-r-----+ 1 root systemd-journal 8388608 Feb 18 07:06 system.journal
/var/log/journal/ae60451991a4be968a3d976e620f44ad:
total 8200
drwxr-sr-x+ 2 root systemd-journal 4096 Feb 18 07:06 .
drwxr-sr-x+ 4 root systemd-journal 4096 Feb 18 07:06 ..
-rw-r-----+ 1 root systemd-journal 8388608 Feb 18 07:06 system.journal
So that just created another log file. Interesting. Let’s check systemd-journald
:
root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# systemctl status systemd-journald
● systemd-journald.service - Journal Service
Loaded: loaded (/lib/systemd/system/systemd-journald.service; static; vendor preset: enabled)
Active: active (running) since Fri 2022-02-18 07:02:25 UTC; 3min 12s ago
TriggeredBy: ● systemd-journald.socket
● systemd-journald-dev-log.socket
● systemd-journald-audit.socket
Docs: man:systemd-journald.service(8)
man:journald.conf(5)
Main PID: 338 (systemd-journal)
Status: "Processing requests..."
Tasks: 1 (limit: 2339)
Memory: 3.9M
CGroup: /system.slice/systemd-journald.service
└─338 /lib/systemd/systemd-journald
root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# ls /var/log/journal/
5392a32a818ae1e9b6e285a07776c843
root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# ls /var/log/journal/*
system.journal
root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# ls /var/log/journal/* -al
total 8204
drwxr-sr-x+ 2 root systemd-journal 4096 Feb 18 07:02 .
drwxr-sr-x+ 3 root systemd-journal 4096 Feb 18 07:02 ..
-rw-r-----+ 1 root systemd-journal 8388608 Feb 18 07:05 system.journal
root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# systemctl restart systemd-journald
root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# ls /var/log/journal/* -al
/var/log/journal/5392a32a818ae1e9b6e285a07776c843:
total 8204
drwxr-sr-x+ 2 root systemd-journal 4096 Feb 18 07:02 .
drwxr-sr-x+ 4 root systemd-journal 4096 Feb 18 07:06 ..
-rw-r-----+ 1 root systemd-journal 8388608 Feb 18 07:06 system.journal
/var/log/journal/ae60451991a4be968a3d976e620f44ad:
total 8200
drwxr-sr-x+ 2 root systemd-journal 4096 Feb 18 07:06 .
drwxr-sr-x+ 4 root systemd-journal 4096 Feb 18 07:06 ..
-rw-r-----+ 1 root systemd-journal 8388608 Feb 18 07:06 system.journal
So it’s running, and I can see the new file being created. Let’s take a look to see if we are seeing logs now…
root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# journalctl -xe
-- Logs begin at Fri 2022-02-18 07:06:46 UTC, end at Fri 2022-02-18 07:06:46 UTC. --
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd[1]: Stopping Journal Service...
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd-journald[338]: Received SIGTERM from PID 1 (systemd).
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd[1]: systemd-journald.service: Succeeded.
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd[1]: Stopped Journal Service.
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd[1]: Starting Journal Service...
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd-journald[2041]: Journal started
-- Subject: The journal has been started
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The system journal process has started up, opened the journal
-- files for writing and is now ready to process requests.
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd-journald[2041]: System Journal (/var/log/journal/ae60451991a4be968a3d976e620f44ad) is 8.0M, max 4.0G, 3.9G free.
-- Subject: Disk space used by the journal
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- System Journal (/var/log/journal/ae60451991a4be968a3d976e620f44ad) is currently using 8.0M.
-- Maximum allowed usage is set to 4.0G.
-- Leaving at least 2.4G free (of currently available 46.7G of disk space).
-- Enforced usage limit is thus 4.0G, of which 3.9G are still available.
--
-- The limits controlling how much disk space is used by the journal may
-- be configured with SystemMaxUse=, SystemKeepFree=, SystemMaxFileSize=,
-- RuntimeMaxUse=, RuntimeKeepFree=, RuntimeMaxFileSize= settings in
-- /etc/systemd/journald.conf. See journald.conf(5) for details.
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd[1]: Started Journal Service.
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd[1]: Starting Flush Journal to Persistent Storage...
-- Subject: A start job for unit systemd-journal-flush.service has begun execution
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit systemd-journal-flush.service has begun execution.
--
-- The job identifier is 1023.
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd-journald[2041]: System Journal (/var/log/journal/ae60451991a4be968a3d976e620f44ad) is 8.0M, max 4.0G, 3.9G free.
-- Subject: Disk space used by the journal
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- System Journal (/var/log/journal/ae60451991a4be968a3d976e620f44ad) is currently using 8.0M.
-- Maximum allowed usage is set to 4.0G.
-- Leaving at least 2.4G free (of currently available 46.7G of disk space).
-- Enforced usage limit is thus 4.0G, of which 3.9G are still available.
--
-- The limits controlling how much disk space is used by the journal may
-- be configured with SystemMaxUse=, SystemKeepFree=, SystemMaxFileSize=,
-- RuntimeMaxUse=, RuntimeKeepFree=, RuntimeMaxFileSize= settings in
-- /etc/systemd/journald.conf. See journald.conf(5) for details.
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd[1]: Finished Flush Journal to Persistent Storage.
-- Subject: A start job for unit systemd-journal-flush.service has finished successfully
So a simple restart of the journal fixed it. Is this an issue on Ubuntu’s side? DigitalOcean’s side? I don’t know. All I know is that I now know nothing about my VMs until I go in there and restart the journal, which was unexpected behavior. (affecting both 20.04 and 18.04)
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.
Hi there,
Thank you for reporting this! Happy to hear that you’ve got it working after the restart!
This should now be fixed with the new Ubuntu base image!
Best,
Bobby