By Alex Garnett
Senior DevOps Technical Writer
In this tutorial, you’ll explore some best practices for keeping your Ubuntu 20.04 server stack up to date. Just as with network security hardening, there are many steps you can take to ensure your servers will continue to work securely without future intervention.
There are a number of tools and configurations that you can apply to most Ubuntu servers beyond what is configured for you automatically. If you’re doing your own server administration, it can be quite disruptive and error-prone to patch all of your environments manually.
This tutorial will cover:
A fundamental part of configuring your server for automatic upgrades is ensuring that all of the applications running on the server are able to restart correctly after unplanned downtime or a reboot. Linux package managers are designed to run non-disruptively in the background so that necessary maintenance does not come with any additional overhead. Despite this, one of the most common reasons for not having a good update strategy in place is being concerned about how your server will behave after being restarted.
Wherever possible, the applications in your stack should be managed by your server’s init system, which on most modern Linux distributions including Ubuntu is systemd. Systemd provides the systemctl command for interacting with your running services and automatically restarting them as needed. Virtually all software that is installed via your package manager and designed to run in the background should automatically provide a systemd service and a configuration unit file as a best practice.
When running your own software, or software deployed from Git repositories, it is not a bad idea to write your own unit files in order to integrate with systemd. As a lightweight alternative, you may prefer to use a tool like supervisor. You can also use your system’s cron scheduler with the
After your configuration is in place, make sure to test it through reboots. You can reboot by running
sudo shutdown now -r, which will cleanly stop your running processes and reboot immediately. You can also specify a time in hh:mm, or a number of minutes from now, instead of
now in order to schedule a reboot in the future. Production deployments should generally not require your attention after any unplanned outages, and all necessary services and endpoints should come back up automatically.
Now that you’ve ensured that your environment won’t have any trouble persisting through maintenance reboots, in the next step, you’ll learn how to schedule automatic updates.
Ubuntu’s package manager,
apt, has a well-established workflow for performing a full system upgrade. First, you run
apt update to refresh your package listings, and then, you run
apt upgrade without specifying a package in order to upgrade every package on the system. This workflow may be slightly different if you have any version conflicts from third-party packages or if you are intentionally keeping some packages unupgraded, but the core commands are the same.
Ubuntu provides a unique tool called
unattended-upgrades in order to automatically retrieve and install security patches and other essential upgrades for your server. Most Ubuntu servers come with this tool automatically installed and configured, but you can install it with the following
- sudo apt update
- sudo apt install unattended-upgrades
After installation, you can check to ensure that the
unattended-upgrades service is running using
- sudo systemctl status unattended-upgrades.service
Output● unattended-upgrades.service - Unattended Upgrades Shutdown
Loaded: loaded (/lib/systemd/system/unattended-upgrades.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2022-02-14 17:51:49 UTC; 3h 4min ago
Main PID: 829 (unattended-upgr)
Tasks: 2 (limit: 1137)
The default configuration of
unattended-upgrades will automatically retrieve bug fix and security updates for most of the packages included in the Ubuntu repositories. However, if you are using older versions of some packages in order to avoid upstream changes, or if your server uses third party package repositories in addition to Ubuntu’s, you can further configure the
Its configuration is stored in
/etc/apt/apt.conf.d/50unattended-upgrades. Open this file using
nano or your favorite text editor:
- sudo nano /etc/apt/apt.conf.d/50unattended-upgrades
The file is well-annotated, and you can see many lines of code comments (beginning with
//) explaining its functionality. The first configuration block handles which packages will be automatically updated, matching a template for Ubuntu package repository names. Files in the core repository and in the
-security repository will be updated by default, but the lines containing the
-backports repositories are commented out by default.
These repositories are disabled by default because they are more likely to contain breaking changes to your installed packages. In order to enable them for unattended upgrades manually, you can delete the
// comment symbols from these lines.
// Automatically upgrade packages from these (origin:archive) pairs
// Note that in Ubuntu security updates may pull in new dependencies
// from non-security sources (e.g. chromium). By allowing the release
// pocket these get automatically pulled in.
// Extended Security Maintenance; doesn't necessarily exist for
// every release and this system may not have it installed, but if
// available, the policy for updates is such that unattended-upgrades
// should also install from here by default.
Further down the file, there are a number of options with
false configuration toggles. For example, there is a toggle to automatically reboot following the installation of packages that need a reboot in order to take effect. You could enable this option by removing the
// comment symbol and changing
true. However, doing so will cause your server to become unavailable at unpredictable intervals. If you enable this option be sure your applications or users can tolerate downtime.
// Automatically reboot *WITHOUT CONFIRMATION* if
// the file /var/run/reboot-required is found after the upgrade
Save and close the file when you are done editing it. If you are using
Ctrl+X, then when prompted,
Y and then Enter.
If you made changes to the configuration, reload the
unattended-upgrades service in order for it to take effect:
- sudo systemctl reload unattended-upgrades.service
You should now have solutions in place to ensure all of the packages on your server receive essential security updates without any additional intervention. In the last step, you’ll learn how to keep your kernel updated, and how best to handle server reboots when they are necessary.
Less often than other packages, you will need to update your system’s kernel. The Linux kernel contains (almost) all running hardware drivers and is responsible for most low-level system interactions. Kernel updates are usually only necessary if there is a high-profile vulnerability to address, if you need to make use of a publicized new kernel feature, or if your kernel has become so old that there is a greater risk of accumulated bugs and vulnerabilities that you may not be aware of.
There is no universal method of automatically scheduling Linux kernel updates. This is because kernel updates have historically required a full system reboot, and scheduling reboots is impossible without making assumptions about your environment. Many servers are expected to provide as close to 24/7 availability as possible, and a reboot can take an unknown amount of time, or require manual intervention.
If you are willing to tolerate some downtime, updating your kernel is straightforward: your unattended
apt updates can be configured to install and prepare new kernels along with other packages, and after rebooting, your server should automatically use the new kernel. Most production deployments require additional complexity around rebooting like this to ensure service availability. For example, you might use a load balancer to automatically redirect traffic to servers that can provide identical functionality in a horizontally scaled deployment while they are individually rebooted in sequence, in order to avoid any visible downtime.
To avoid downtime during kernel upgrades, you can use a feature of the Linux kernel called live patching. This feature makes it possible to implement kernel updates without rebooting. There are two major maintainers for kernel live patches: Canonical, who provides their own Livepatch Service for Ubuntu, and KernelCare who support Ubuntu in addition to most other major Linux distributions. Both require registration to use, and only Canonical’s service is free for individual use.
You can register for a Livepatch key at https://auth.livepatch.canonical.com/. After enrolling, you can install the
canonical-livepatch snap package. Snap is another Ubuntu package manager that runs alongside
- sudo snap install canonical-livepatch
You can enable
canonical-livepatch with a one-line command using the key from their website:
- sudo canonical-livepatch enable your-key
The output should contain the message
Successfully enabled device. The service should run in the background from now on without any further intervention, and you can check its status using
- sudo canonical-livepatch status
Outputlast check: 55 seconds ago
server check-in: succeeded
patch state: ✓ all applicable livepatch modules inserted
patch version: 84.1
tier: updates (Free usage; This machine beta tests new patches.)
machine id: d56589e7fa994005a266d4caf9b9dcf7
You have now configured automatic kernel updates for your server, meaning it should no longer be necessary to reboot in order to maintain a secure and up-to-date environment.
In this tutorial, you explored multiple strategies to keep your Ubuntu servers updated automatically. You also learned some of the nuances of package repositories, kernel updates, and handling server reboots. These are all important fundamentals of DevOps and of working with servers more broadly, and nearly all production configurations build on these core concepts.
Next, you may want to learn more about Package Management in Ubuntu. If you would like to learn more about the Snap package format, visit our tutorial How To Package and Publish a Snap Application on Ubuntu 18.04 .
Thanks for learning with the DigitalOcean Community. Check out our offerings for compute, storage, networking, and managed databases.
Working on improving health and education, reducing inequality, and spurring economic growth? We'd like to help.