The author selected the Free and Open Source Fund to receive a donation as part of the Write for DOnations program.
System logs are an extremely important component of managing Linux systems. They provide an invaluable insight into how the systems are working and also how they are being used because, in addition to errors, they record operational information such as security events. The standard configuration for Linux systems is to store their logs locally on the same system where they occurred. This works for standalone systems but quickly becomes a problem as the number of systems increases. The solution to managing all these logs is to create a centralized logging server where each Linux host sends its logs, in real-time, to a dedicated log management server.
A centralized logging solution offers several benefits compared with storing logs on each host:
In this guide, you will configure a component of the systemd suite of tools to relay log messages from client systems to a centralized log collection server. You will configure the server and client to use TLS certificates to encrypt the log messages as they are transmitted across insecure networks such as the internet and also to authenticate each other.
Before you begin this guide you’ll need the following:
This guide will use the following two example hostnames:
client.your_domain
: The client system that generates the logs.server.your_domain
: The log collection server.Log in to both the client and server in separate terminals via SSH as the non-root sudo user to begin this tutorial.
Note: Throughout the tutorial, command blocks are labeled with the server name (client or server) that the command should be run on.
systemd-journal-remote
In this step, you will install the systemd-journal-remote
package on the client and the server. This package contains the components that the client and server use to relay log messages.
First, on both the client and server, run a system update to ensure that the package database and the system is current:
- sudo apt update
- sudo apt upgrade
Next, install the systemd-journal-remote
package:
- sudo apt install systemd-journal-remote
On the server, enable and start the two systemd
components that it needs to receive log messages with the following command:
- sudo systemctl enable --now systemd-journal-remote.socket
- sudo systemctl enable systemd-journal-remote.service
The --now
option in the first command starts the services immediately. You did not use it in the second command because this service will not start until it has TLS certificates, which you will create in the next step.
On the client, enable the component that systemd
uses to send the log messages to the server:
- sudo systemctl enable systemd-journal-upload.service
Next, on the server, open ports 19532
and 80
in the UFW firewall. This will allow the server to receive the log messages from the client. Port 80
is the port that certbot
will use to generate the TLS certificate. The following commands will open these ports:
- sudo ufw allow in 19532/tcp
- sudo ufw allow in 80/tcp
On the client, you only need to open port 80
with this command:
- sudo ufw allow in 80/tcp
You have now installed the required components and completed the base system configuration on the client and server. Before you can configure these components to start relaying log messages you will register the Let’s Encrypt TLS certificates for the client and server using the certbot
utility.
Let’s Encrypt is a Certificate Authority that issues free TLS certificates. These certificates allow computers to both encrypt the data that they send between them and also verify each other’s identity. These certificates are what allow you to secure your internet browsing with HTTPS. The same certificates can be used by any other application that wants the same level of security. The process of registering the certificate is the same no matter what you use them for.
In this step, you will install the certbot
utility and use it to register the certificates. It will also automatically take care of renewing the certificates when they expire. The registration process here is the same on the client and server. You only need to change the hostname to match the host where you are running the registration command.
First, install certbot
and the curl
utility on both hosts:
- sudo apt install certbot curl
Now you’ve installed certbot
, run the following command to register the certificates on the client and server:
- sudo certbot certonly --standalone --agree-tos --email sammy@your_domain -d your_domain
The options in this command mean as follows:
certonly
: Register the certificate and make no other changes on the system.--standalone
: Use certbot’s built-in web server to verify the certificate request.--agree-tos
: Automatically agree to the Let’s Encrypt Terms of Service.--email your-email
: This is the email address that Let’s Encrypt will use to notify you about certificate expiry and other important information.-d your_domain
: The hostname that the certificate will be registered for. This must match the system where you run it.When you run this command you will be asked if you want to share the email address with Let’s Encrypt so they can email you news and other information about their work. Doing this is optional, if you do not share your email address the certificate registration will still complete normally.
When the certificate registration process completes it will place the certificate and key files in /etc/letsencrypt/live/your_domain/
where your_domain
is the hostname that you registered the certificate for.
Finally, you need to download a copy of the Let’s Encrypt CA and intermediate certificates and put them into the same file. journald
will use this file to verify the authenticity of the certificates on the client and server when they communicate with each other.
The following command will download the two certificates from the Let’s Encrypt website and put them into a single file called letsencrypt-combined-certs.pem
in your user’s home directory.
Run this command on the client and server to download the certificates and create the combined file:
- curl -s https://letsencrypt.org/certs/{isrgrootx1.pem.txt,letsencryptauthorityx3.pem.txt} > ~/letsencrypt-combined-certs.pem
Next, move this file into the Let’s Encrypt directory containing the certificates and keys:
- sudo cp ~/letsencrypt-combined-certs.pem /etc/letsencrypt/live/your_domain/
You’ve now registered the certificates and keys. In the next step, you will configure the log collection server to start listening for and storing log messages from the client.
In this step, you will configure the server to use the certificate and key files that you generated in the last step so that it can start accepting log messages from the client.
systemd-journal-remote
is the component that listens for log messages. Open its configuration file at /etc/systemd/journal-remote.conf
with a text editor to start configuring it on the server:
- sudo nano /etc/systemd/journal-remote.conf
Next, uncomment all the lines under the [Remote]
section and set the paths to point to the TLS files you just created:
[Remote]
Seal=false
SplitMode=host
ServerKeyFile=/etc/letsencrypt/live/server.your_domain/privkey.pem
ServerCertificateFile=/etc/letsencrypt/live/server.your_domain/fullchain.pem
TrustedCertificateFile=/etc/letsencrypt/live/server.your_domain/letsencrypt-combined-certs.pem
Here are the options you’ve used here:
Seal=false
: Sign the log data in the journal. Enable this if you need maximum security; otherwise, you can leave it as false
.SplitMode=host
: The logs from the remote clients will be split by host in /var/log/journal/remote
. If you would prefer all the logs to be added to a single file set this to SplitMode=false
.ServerKeyFile
: The server’s private key file.ServerCertificateFile
: The server’s certificate file.TrustedCertificateFile
: The file containing the Let’s Encrypt CA certificates.Now, you need to change the permissions on the Let’s Encrypt directories that contain the certificates and key so that the systemd-journal-remote
can read and use them.
First, change the permissions so that the certificate and private key are readable:
- sudo chmod 0755 /etc/letsencrypt/{live,archive}
- sudo chmod 0640 /etc/letsencrypt/live/server.your_domain/privkey.pem
Next, change the group ownership of the private key to systemd-journal-remote
’s group:
- sudo chgrp systemd-journal-remote /etc/letsencrypt/live/server.your_domain/privkey.pem
You can now start systemd-journal-remote
:
- sudo systemctl start systemd-journal-remote.service
Your log collection server is now running and ready to start accepting log messages from a client. In the next step, you will configure the client to relay the logs to your collection server.
In this step, you will configure the component that relays the log messages to the log collection server. This component is called systemd-journal-upload
.
The default configuration for systemd-journal-upload
is that it uses a temporary user that only exists while the process is running. This makes allowing systemd-journal-upload
to read the TLS certificates and keys more complicated. To resolve this you will create a new system user with the same name as the temporary user that will get used in its place.
First, create the new user called systemd-journal-upload
on the client with the following adduser
command:
- sudo adduser --system --home /run/systemd --no-create-home --disabled-login --group systemd-journal-upload
These options to the command are:
--system
: Create the new user as a system user. This gives the user a UID (User Identifier) number under 1000
. UID’s over 1000
are usually given to user accounts that a human will use to log in with.--home /run/systemd
: Set /run/systemd
as the home directory for this user.--no-create-home
: Don’t create the home directory set, as it already exists.--disabled-login
: The user cannot log in to the server via, for example, SSH.--group
: Create a group with the same name as the user.Next, set the permissions and ownership of the Let’s Encrypt certificate files:
- sudo chmod 0755 /etc/letsencrypt/{live,archive}
- sudo chmod 0640 /etc/letsencrypt/live/client.your_domain/privkey.pem
- sudo chgrp systemd-journal-upload /etc/letsencrypt/live/client.your_domain/privkey.pem
Now, edit the configuration for systemd-journal-upload
, which is at /etc/systemd/journal-upload.conf
. Open this file with a text editor:
- sudo nano /etc/systemd/journal-upload.conf
Edit this file so that it looks like the following:
[Upload]
URL=https://server.your_domain:19532
ServerKeyFile=/etc/letsencrypt/live/client.your_domain/privkey.pem
ServerCertificateFile=/etc/letsencrypt/live/client.your_domain/fullchain.pem
TrustedCertificateFile=/etc/letsencrypt/live/client.your_domain/letsencrypt-combined-certs.pem
Finally, restart the systemd-journal-upload
service so it uses the new configuration:
- sudo systemctl restart systemd-journal-upload.service
Your client is now set up and running and is sending its log messages to the log collection server. In the next step, you will check that the logs are being sent and recorded correctly.
In this step, you will test that the client is relaying log messages to the server and that the server is storing them correctly.
The log collection server stores the logs from the clients in a directory at /var/log/journal/remote/
. When you restarted the client at the end of the last step it began sending log messages so there is now a log file in /var/log/journal/remote/
. The file will be named after the hostname you used for the TLS certificate.
Use the ls
command to check that the client’s log file is present on the server:
- sudo ls -la /var/log/journal/remote/
This will print the directory contents showing the log file:
Outputtotal 16620
drwxr-xr-x 2 systemd-journal-remote systemd-journal-remote 4096 Jun 30 16:17 .
drwxr-sr-x+ 4 root systemd-journal 4096 Jun 30 15:55 ..
-rw-r----- 1 systemd-journal-remote systemd-journal-remote 8388608 Jul 1 10:46 'remote-CN=client.your_domain'
Next, write a log message on the client to check that the server is receiving the client’s messages as you expect. You will use the logger utility to create a custom log message on the client. If everything is working systemd-journal-upload
will relay this message to the server.
On the client run the following logger
command:
- sudo logger -p syslog.debug "### TEST MESSAGE from client.your_domain ###"
The -p syslog.debug
in this command sets the facility and severity of the message. Setting this to syslog.debug
will make clear it’s a test message. This command will record the message ### TEST MESSAGE from client.your_domain ###
to the client’s journal, which systemd-journal-upload
then relays to the server.
Next, read the client’s journal file on the server to check that the log messages are arriving from the client. This file is a binary log file so you will not be able to read it with tools like less
. Instead, read the file using journalctl
with the --file=
option that allows you to specify a custom journal file:
- sudo journalctl --file=/var/log/journal/remote/remote-CN=client.your_domain.journal
The log message will appear as follows:
Test log message. . .
Jun 29 13:10:09 client root[3576]: ### TEST MESSAGE from client.your_domain ###
Your log centralization server is now successfully collecting logs from your client system.
In this article, you set up a log central collection server and configured a client to relay a copy of its system logs to the server. You can configure as many clients as you need to relay messages to the log collection server using the client configuration steps you used here.
Thanks for learning with the DigitalOcean Community. Check out our offerings for compute, storage, networking, and managed databases.
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
Sign up for Infrastructure as a Newsletter.
Working on improving health and education, reducing inequality, and spurring economic growth? We'd like to help.
Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.
I ran into an issue when setting
SplitMode=false
as described in the instructions. This produces error in Version249.11-0ubuntu3.6
I checked the documentation of the service here: https://www.freedesktop.org/software/systemd/man/journal-remote.conf.html.
It sais value should be: “host” or “none”. With the latter it works fine. Instructions should be updated.