Ubuntu Journal is broken on boot

100% reproducible:

Create Droplet -> Ubuntu 20.04

As root:

# journalctl -xe
-- No entries --

Well THAT’S CONCERNING. Let’s see if it’s actually there:

root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# ls /var/log/journal/* -al                                                    
total 8204                                                
drwxr-sr-x+ 2 root systemd-journal    4096 Feb 18 07:02 .
drwxr-sr-x+ 3 root systemd-journal    4096 Feb 18 07:02 ..                                                          
-rw-r-----+ 1 root systemd-journal 8388608 Feb 18 07:05 system.journal

So, the system.journal file is there, but it isn’t being recognized by journalctl? Weird.

Let’s try turning it off and back on again…

root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# systemctl restart systemd-journald                                           
root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# ls /var/log/journal/* -al                                                                                                                                                                         
/var/log/journal/5392a32a818ae1e9b6e285a07776c843:                                                                  
total 8204                                  
drwxr-sr-x+ 2 root systemd-journal    4096 Feb 18 07:02 .                                                           
drwxr-sr-x+ 4 root systemd-journal    4096 Feb 18 07:06 ..                                                          
-rw-r-----+ 1 root systemd-journal 8388608 Feb 18 07:06 system.journal                                              
                                                                                                                                                                                                                                         
/var/log/journal/ae60451991a4be968a3d976e620f44ad:                                                                                                                                                                                       
total 8200                                                                                                          
drwxr-sr-x+ 2 root systemd-journal    4096 Feb 18 07:06 . 
drwxr-sr-x+ 4 root systemd-journal    4096 Feb 18 07:06 ..                                                          
-rw-r-----+ 1 root systemd-journal 8388608 Feb 18 07:06 system.journal                                              

So that just created another log file. Interesting. Let’s check systemd-journald:

root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# systemctl status systemd-journald
● systemd-journald.service - Journal Service
     Loaded: loaded (/lib/systemd/system/systemd-journald.service; static; vendor preset: enabled)
     Active: active (running) since Fri 2022-02-18 07:02:25 UTC; 3min 12s ago
TriggeredBy: ● systemd-journald.socket                                                                              
             ● systemd-journald-dev-log.socket            
             ● systemd-journald-audit.socket                                                                        
       Docs: man:systemd-journald.service(8)                                                                        
             man:journald.conf(5)                        
   Main PID: 338 (systemd-journal)                                                                                  
     Status: "Processing requests..."                                                                               
      Tasks: 1 (limit: 2339)                                                                                        
     Memory: 3.9M                                                                                                   
     CGroup: /system.slice/systemd-journald.service                                                                 
             └─338 /lib/systemd/systemd-journald                                                                    
root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# ls /var/log/journal/                                    
5392a32a818ae1e9b6e285a07776c843                                                                                    
root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# ls /var/log/journal/*                          
system.journal                                                                                                      
root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# ls /var/log/journal/* -al                                                    
total 8204            
drwxr-sr-x+ 2 root systemd-journal    4096 Feb 18 07:02 .
drwxr-sr-x+ 3 root systemd-journal    4096 Feb 18 07:02 ..                                                                                                                                                                               
-rw-r-----+ 1 root systemd-journal 8388608 Feb 18 07:05 system.journal
root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# systemctl restart systemd-journald
root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# ls /var/log/journal/* -al                                                                                                       
/var/log/journal/5392a32a818ae1e9b6e285a07776c843:
total 8204                                                                                                          
drwxr-sr-x+ 2 root systemd-journal    4096 Feb 18 07:02 .
drwxr-sr-x+ 4 root systemd-journal    4096 Feb 18 07:06 ..                                                          
-rw-r-----+ 1 root systemd-journal 8388608 Feb 18 07:06 system.journal                        
                                                          
/var/log/journal/ae60451991a4be968a3d976e620f44ad:                                                                  
total 8200                                                                                                          
drwxr-sr-x+ 2 root systemd-journal    4096 Feb 18 07:06 .                                                           
drwxr-sr-x+ 4 root systemd-journal    4096 Feb 18 07:06 ..                                                          
-rw-r-----+ 1 root systemd-journal 8388608 Feb 18 07:06 system.journal  

So it’s running, and I can see the new file being created. Let’s take a look to see if we are seeing logs now…

root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# journalctl -xe                                                               
-- Logs begin at Fri 2022-02-18 07:06:46 UTC, end at Fri 2022-02-18 07:06:46 UTC. --
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd[1]: Stopping Journal Service...
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd-journald[338]: Received SIGTERM from PID 1 (systemd).
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd[1]: systemd-journald.service: Succeeded.
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd[1]: Stopped Journal Service.
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd[1]: Starting Journal Service...
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd-journald[2041]: Journal started
-- Subject: The journal has been started
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- The system journal process has started up, opened the journal
-- files for writing and is now ready to process requests.
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd-journald[2041]: System Journal (/var/log/journal/ae60451991a4be968a3d976e620f44ad) is 8.0M, max 4.0G, 3.9G free.
-- Subject: Disk space used by the journal
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- System Journal (/var/log/journal/ae60451991a4be968a3d976e620f44ad) is currently using 8.0M.
-- Maximum allowed usage is set to 4.0G.
-- Leaving at least 2.4G free (of currently available 46.7G of disk space).
-- Enforced usage limit is thus 4.0G, of which 3.9G are still available.
-- 
-- The limits controlling how much disk space is used by the journal may
-- be configured with SystemMaxUse=, SystemKeepFree=, SystemMaxFileSize=,
-- RuntimeMaxUse=, RuntimeKeepFree=, RuntimeMaxFileSize= settings in
-- /etc/systemd/journald.conf. See journald.conf(5) for details.
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd[1]: Started Journal Service.
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd[1]: Starting Flush Journal to Persistent Storage...
-- Subject: A start job for unit systemd-journal-flush.service has begun execution
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- A start job for unit systemd-journal-flush.service has begun execution.
-- 
-- The job identifier is 1023.
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd-journald[2041]: System Journal (/var/log/journal/ae60451991a4be968a3d976e620f44ad) is 8.0M, max 4.0G, 3.9G free.
-- Subject: Disk space used by the journal
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- System Journal (/var/log/journal/ae60451991a4be968a3d976e620f44ad) is currently using 8.0M.
-- Maximum allowed usage is set to 4.0G.
-- Leaving at least 2.4G free (of currently available 46.7G of disk space).
-- Enforced usage limit is thus 4.0G, of which 3.9G are still available.
-- 
-- The limits controlling how much disk space is used by the journal may
-- be configured with SystemMaxUse=, SystemKeepFree=, SystemMaxFileSize=,
-- RuntimeMaxUse=, RuntimeKeepFree=, RuntimeMaxFileSize= settings in
-- /etc/systemd/journald.conf. See journald.conf(5) for details.
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd[1]: Finished Flush Journal to Persistent Storage.
-- Subject: A start job for unit systemd-journal-flush.service has finished successfully

So a simple restart of the journal fixed it. Is this an issue on Ubuntu’s side? DigitalOcean’s side? I don’t know. All I know is that I now know nothing about my VMs until I go in there and restart the journal, which was unexpected behavior. (affecting both 20.04 and 18.04)