By Alex Garnett
Senior DevOps Technical Writer
In this tutorial, you’ll explore some best practices for keeping your Rocky Linux 8 server stack up to date. Just as with network security hardening, there are many steps you can take to ensure your servers will continue to work securely without future intervention.
There are a number of tools and configurations that you can apply to most Rocky Linux servers beyond what is configured for you automatically. If you’re doing your own server administration, it can be quite disruptive and error-prone to patch all of your environments manually.
This tutorial will cover:
A fundamental part of configuring your server for automatic upgrades is ensuring that all of the applications running on the server are able to restart correctly after unplanned downtime or a reboot. Linux package managers are designed to run non-disruptively in the background so that necessary maintenance does not come with any additional overhead. Despite this, one of the most common reasons for not having a good update strategy in place is being concerned about how your server will behave after being restarted.
Wherever possible, the applications in your stack should be managed by your server’s init system, which on most modern Linux distributions including Rocky Linux is systemd. Systemd provides the systemctl command for interacting with your running services and automatically restarting them as needed. Virtually all software that is installed via your package manager and designed to run in the background should automatically provide a systemd service and a configuration unit file as a best practice.
When running your own software, or software deployed from Git repositories, it is not a bad idea to write your own unit files to integrate with systemd. As a lightweight alternative, you may prefer to use a tool like supervisor. You can also use your system’s cron scheduler with the
After your configuration is in place, make sure to test it through reboots. You can reboot by running
sudo shutdown now -r, which will cleanly stop your running processes and reboot immediately. You can also specify a time in hh:mm, or a number of minutes from now, instead of
now to schedule a reboot in the future. Production deployments should generally not require your attention after any unplanned outages, and all necessary services and endpoints should come back up automatically.
Now that you’ve ensured that your environment won’t have any trouble persisting through maintenance reboots, in the next step, you’ll learn how to schedule automatic upgrades.
Rocky’s package manager,
dnf, has two main ways of performing a full system upgrade. You can run
dnf upgrade without specifying a package to upgrade every package on your system. You can also run
dnf upgrade-minimal to upgrade every package only to the latest bugfix or security patch release, which will perform necessary maintenance while avoiding any potential breaking changes upstream. You can read more about
dnf commands from the dnf documentation.
Rocky also provides a unique tool called
dnf-automatic to automatically retrieve and install security patches and other essential upgrades for your server. You can install it using
- sudo dnf install dnf-automatic -y
dnf-automatic is not enabled automatically after installation. Instead, it provides several different services that you can register with Systemd to customize its behavior. These are:
dnf-automaticconfiguration file options in
For this tutorial, you will be enabling the
dnf-automatic-install service, but first, you’ll make one change to the
dnf-automatic configuration file. Open the file using
vi or your favorite text editor:
- sudo vi /etc/dnf/automatic
[commands] # What kind of upgrade to perform: # default = all available upgrades # security = only the security upgrades upgrade_type = security random_sleep = 0 # Maximum time in seconds to wait until the system is on-line and able to # connect to remote repositories. network_online_timeout = 60 # To just receive updates use dnf-automatic-notifyonly.timer # Whether updates should be downloaded when they are available, by # dnf-automatic.timer. notifyonly.timer, download.timer and # install.timer override this setting. download_updates = yes # Whether updates should be applied when they are available, by # dnf-automatic.timer. notifyonly.timer, download.timer and # install.timer override this setting. apply_updates = no …
Most of the options in this file correspond to the various overrides provided by the different Systemd services. An exception is the
upgrade_type option, which is set to
default by default. If you’re going to be implementing automatic upgrades (the most proactive configuration), it’s a good idea to only install security upgrades by default, to avoid unexpected changes in functionality. Change the
upgrade_type value to
security as in the above example, then save and close the file. If you are using
vi, you can save and quit by entering
Now you can enable the service using
- sudo systemctl enable dnf-automatic-install.timer
You can check to ensure that the
dnf-automatic-install service is running correctly using
- sudo systemctl status dnf-automatic-install
Output● dnf-automatic-install.service - dnf automatic install updates Loaded: loaded (/usr/lib/systemd/system/dnf-automatic-install.service; static; vendor preset: disabled) Active: inactive (dead) Jul 14 21:01:03 droplet-name dnf-automatic: No security updates needed, but 154 updates available Jul 14 21:01:03 droplet-name systemd: dnf-automatic-install.service: Succeeded. Jul 14 21:01:03 droplet-name systemd: Started dnf automatic install updates.
Unlike some Systemd services,
dnf-automatic is implemented as a timer, rather than as a service that runs perpetually in the background. This means that the status of
Active: inactive (dead) is expected between activations, as long as the service is loaded and logs reflect successful runs.
You can check on the details of that timer using
- sudo systemctl cat dnf-automatic-install.timer
Output[Unit] Description=dnf-automatic-install timer # See comment in dnf-makecache.service ConditionPathExists=!/run/ostree-booted Wants=network-online.target [Timer] OnCalendar=*-*-* 6:00 RandomizedDelaySec=60m Persistent=true [Install] WantedBy=timers.target
By default the service is configured to run around 6:00 every day. You should not need to change this value.
You should now have solutions in place to ensure all of the packages on your server receive essential security updates without any additional intervention. In the last step, you’ll learn how to keep your kernel updated, and how best to handle server reboots when they are necessary.
Less often than other packages, you will need to update your system’s kernel. The Linux kernel contains (almost) all running hardware drivers and is responsible for most low-level system interactions. Kernel updates are usually only necessary if there is a high-profile vulnerability to address, if you need to make use of a publicized new kernel feature, or if your kernel has become so old that there is a greater risk of accumulated bugs and vulnerabilities that you may not be aware of.
There is no universal method of automatically scheduling Linux kernel updates. This is because kernel updates have historically required a full system reboot, and scheduling reboots is impossible without making assumptions about your environment. Many servers are expected to provide as close to 24/7 availability as possible, and a reboot can take an unknown amount of time, or require manual intervention.
Most production deployments require additional complexity around rebooting like this to ensure service availability. For example, you might use a load balancer to automatically redirect traffic to servers that can provide identical functionality in a horizontally scaled deployment while they are individually rebooted in sequence, to avoid any visible downtime.
To avoid downtime during kernel upgrades, you can use a feature of the Linux kernel called live patching. This feature makes it possible to implement kernel updates without rebooting. There are two major maintainers for kernel live patches in the Rocky Linux ecosystem: Red Hat’s
kpatch, which provides live patching for Red Hat Enterprise Linux, and KernelCare who support Rocky Linux in addition to most other major Linux distributions. Both require registration to use.
Due to the way that Red Hat Enterprise Linux’s licensing model works, you will not be able to receive live kernel patches via
kpatch when running Rocky Linux — Rocky acts in many ways like an unlicensed version of Red Hat, and there is no way to enroll in kernel live patching without running a full licensed Red Hat instance. However, the method to enable it is the same on Rocky or RHEL. First, install the
- sudo dnf install kpatch-dnf
dnf kpatch auto to automatically subscribe to live patching services:
- sudo dnf kpatch auto
OutputLast metadata expiration check: 0:00:06 ago on Thu 14 Jul 2022 09:12:07 PM UTC. Dependencies resolved. Nothing to do. Complete!
kpatch will check for configured patching services, and, finding none, will exit gracefully. On Red Hat, this would instead perform a license check and register with the Red Hat
kpatch servers. On Rocky, you may want to investigate KernelCare as a commercial support option instead.
In this tutorial, you explored multiple strategies to keep your Rocky Linux servers updated automatically. You also learned some of the nuances of package repositories, kernel updates, and handling server reboots. These are all important fundamentals of DevOps and of working with servers more broadly, and nearly all production configurations build on these core concepts.
Next, you may want to learn to use Watchtower in order to automatically update Docker container images.
Thanks for learning with the DigitalOcean Community. Check out our offerings for compute, storage, networking, and managed databases.