Question

Ubuntu Journal is broken on boot

100% reproducible:

Create Droplet -> Ubuntu 20.04

As root:

# journalctl -xe
-- No entries --

Well THAT’S CONCERNING. Let’s see if it’s actually there:

root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# ls /var/log/journal/* -al                                                    
total 8204                                                
drwxr-sr-x+ 2 root systemd-journal    4096 Feb 18 07:02 .
drwxr-sr-x+ 3 root systemd-journal    4096 Feb 18 07:02 ..                                                          
-rw-r-----+ 1 root systemd-journal 8388608 Feb 18 07:05 system.journal

So, the system.journal file is there, but it isn’t being recognized by journalctl? Weird.

Let’s try turning it off and back on again…

root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# systemctl restart systemd-journald                                           
root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# ls /var/log/journal/* -al                                                                                                                                                                         
/var/log/journal/5392a32a818ae1e9b6e285a07776c843:                                                                  
total 8204                                  
drwxr-sr-x+ 2 root systemd-journal    4096 Feb 18 07:02 .                                                           
drwxr-sr-x+ 4 root systemd-journal    4096 Feb 18 07:06 ..                                                          
-rw-r-----+ 1 root systemd-journal 8388608 Feb 18 07:06 system.journal                                              
                                                                                                                                                                                                                                         
/var/log/journal/ae60451991a4be968a3d976e620f44ad:                                                                                                                                                                                       
total 8200                                                                                                          
drwxr-sr-x+ 2 root systemd-journal    4096 Feb 18 07:06 . 
drwxr-sr-x+ 4 root systemd-journal    4096 Feb 18 07:06 ..                                                          
-rw-r-----+ 1 root systemd-journal 8388608 Feb 18 07:06 system.journal                                              

So that just created another log file. Interesting. Let’s check systemd-journald:

root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# systemctl status systemd-journald
● systemd-journald.service - Journal Service
     Loaded: loaded (/lib/systemd/system/systemd-journald.service; static; vendor preset: enabled)
     Active: active (running) since Fri 2022-02-18 07:02:25 UTC; 3min 12s ago
TriggeredBy: ● systemd-journald.socket                                                                              
             ● systemd-journald-dev-log.socket            
             ● systemd-journald-audit.socket                                                                        
       Docs: man:systemd-journald.service(8)                                                                        
             man:journald.conf(5)                        
   Main PID: 338 (systemd-journal)                                                                                  
     Status: "Processing requests..."                                                                               
      Tasks: 1 (limit: 2339)                                                                                        
     Memory: 3.9M                                                                                                   
     CGroup: /system.slice/systemd-journald.service                                                                 
             └─338 /lib/systemd/systemd-journald                                                                    
root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# ls /var/log/journal/                                    
5392a32a818ae1e9b6e285a07776c843                                                                                    
root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# ls /var/log/journal/*                          
system.journal                                                                                                      
root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# ls /var/log/journal/* -al                                                    
total 8204            
drwxr-sr-x+ 2 root systemd-journal    4096 Feb 18 07:02 .
drwxr-sr-x+ 3 root systemd-journal    4096 Feb 18 07:02 ..                                                                                                                                                                               
-rw-r-----+ 1 root systemd-journal 8388608 Feb 18 07:05 system.journal
root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# systemctl restart systemd-journald
root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# ls /var/log/journal/* -al                                                                                                       
/var/log/journal/5392a32a818ae1e9b6e285a07776c843:
total 8204                                                                                                          
drwxr-sr-x+ 2 root systemd-journal    4096 Feb 18 07:02 .
drwxr-sr-x+ 4 root systemd-journal    4096 Feb 18 07:06 ..                                                          
-rw-r-----+ 1 root systemd-journal 8388608 Feb 18 07:06 system.journal                        
                                                          
/var/log/journal/ae60451991a4be968a3d976e620f44ad:                                                                  
total 8200                                                                                                          
drwxr-sr-x+ 2 root systemd-journal    4096 Feb 18 07:06 .                                                           
drwxr-sr-x+ 4 root systemd-journal    4096 Feb 18 07:06 ..                                                          
-rw-r-----+ 1 root systemd-journal 8388608 Feb 18 07:06 system.journal  

So it’s running, and I can see the new file being created. Let’s take a look to see if we are seeing logs now…

root@ubuntu-s-1vcpu-2gb-amd-nyc3-01:~# journalctl -xe                                                               
-- Logs begin at Fri 2022-02-18 07:06:46 UTC, end at Fri 2022-02-18 07:06:46 UTC. --
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd[1]: Stopping Journal Service...
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd-journald[338]: Received SIGTERM from PID 1 (systemd).
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd[1]: systemd-journald.service: Succeeded.
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd[1]: Stopped Journal Service.
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd[1]: Starting Journal Service...
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd-journald[2041]: Journal started
-- Subject: The journal has been started
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- The system journal process has started up, opened the journal
-- files for writing and is now ready to process requests.
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd-journald[2041]: System Journal (/var/log/journal/ae60451991a4be968a3d976e620f44ad) is 8.0M, max 4.0G, 3.9G free.
-- Subject: Disk space used by the journal
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- System Journal (/var/log/journal/ae60451991a4be968a3d976e620f44ad) is currently using 8.0M.
-- Maximum allowed usage is set to 4.0G.
-- Leaving at least 2.4G free (of currently available 46.7G of disk space).
-- Enforced usage limit is thus 4.0G, of which 3.9G are still available.
-- 
-- The limits controlling how much disk space is used by the journal may
-- be configured with SystemMaxUse=, SystemKeepFree=, SystemMaxFileSize=,
-- RuntimeMaxUse=, RuntimeKeepFree=, RuntimeMaxFileSize= settings in
-- /etc/systemd/journald.conf. See journald.conf(5) for details.
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd[1]: Started Journal Service.
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd[1]: Starting Flush Journal to Persistent Storage...
-- Subject: A start job for unit systemd-journal-flush.service has begun execution
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- A start job for unit systemd-journal-flush.service has begun execution.
-- 
-- The job identifier is 1023.
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd-journald[2041]: System Journal (/var/log/journal/ae60451991a4be968a3d976e620f44ad) is 8.0M, max 4.0G, 3.9G free.
-- Subject: Disk space used by the journal
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- System Journal (/var/log/journal/ae60451991a4be968a3d976e620f44ad) is currently using 8.0M.
-- Maximum allowed usage is set to 4.0G.
-- Leaving at least 2.4G free (of currently available 46.7G of disk space).
-- Enforced usage limit is thus 4.0G, of which 3.9G are still available.
-- 
-- The limits controlling how much disk space is used by the journal may
-- be configured with SystemMaxUse=, SystemKeepFree=, SystemMaxFileSize=,
-- RuntimeMaxUse=, RuntimeKeepFree=, RuntimeMaxFileSize= settings in
-- /etc/systemd/journald.conf. See journald.conf(5) for details.
Feb 18 07:06:46 ubuntu-s-1vcpu-2gb-amd-nyc3-01 systemd[1]: Finished Flush Journal to Persistent Storage.
-- Subject: A start job for unit systemd-journal-flush.service has finished successfully

So a simple restart of the journal fixed it. Is this an issue on Ubuntu’s side? DigitalOcean’s side? I don’t know. All I know is that I now know nothing about my VMs until I go in there and restart the journal, which was unexpected behavior. (affecting both 20.04 and 18.04)


Submit an answer

This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

Sign In or Sign Up to Answer

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

Want to learn more? Join the DigitalOcean Community!

Join our DigitalOcean community of over a million developers for free! Get help and share knowledge in Q&A, subscribe to topics of interest, and get courses and tools that will help you grow as a developer and scale your project or business.

Hi there,

Thank you for reporting this! Happy to hear that you’ve got it working after the restart!

This should now be fixed with the new Ubuntu base image!

Best,

Bobby