HAProxy, which stands for High Availability Proxy, is a widely used TCP and HTTP-based proxy server that runs on Linux, Solaris, and FreeBSD. It is used to load balance applications by distributing requests between multiple servers, and to ensure that applications are highly available for users.
By default on many CentOS 8 installations, HAProxy is not configured to write its log output to a file. This quickstart tutorial will explain how to configure HAProxy logging with Rsyslog by using a Unix domain socket for reliability, speed, and security.
To complete this tutorial, you will need a server running CentOS 8. This server should have a non-root user with administrative privileges. To set this up, follow the Initial Server Setup guide for CentOS 8 tutorial.
To install HAProxy, run the following dnf
command:
- sudo dnf install haproxy
When you are prompted Is this ok [y/N]:
enter y
and press RETURN
to install the package.
Once you have installed HAProxy, enable and start it using the systemctl
command:
- sudo systemctl enable haproxy.service
You should receive the following output:
OutputCreated symlink /etc/systemd/system/multi-user.target.wants/haproxy.service → /usr/lib/systemd/system/haproxy.service.
With the HAProxy service now enabled, you can start it up to ensure that it runs with a default configuration on your server:
- sudo systemctl start haproxy.service
Next examine HAProxy’s status to make sure it is running:
- sudo systemctl status haproxy.service
You should receive output like the following. Note the highlighted active (running)
portion of the output. If your server shows the same highlighted section then HAProxy is running correctly on your server and you can proceed with configuring logging.
Output● haproxy.service - HAProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2020-09-09 21:16:39 UTC; 4min 39s ago
Process: 21340 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q (code=exited, status=0/SUCCESS)
Main PID: 21341 (haproxy)
Tasks: 2 (limit: 2881)
Memory: 2.7M
CGroup: /system.slice/haproxy.service
├─21341 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
. . .
If your output is different, or the status shows something like Active: failed
, then follow the troubleshooting steps in the How to Troubleshoot Common HAProxy Errors tutorial to determine what is preventing HAProxy from starting correctly.
Once you have confirmed that HAProxy is enabled and running, you can continue to the next step, which is configuring HAProxy’s logging directives.
To configure HAProxy’s logging directives, open /etc/haproxy/haproxy.cfg
in vi
or your preferred editor:
- sudo vi /etc/haproxy/haproxy.cfg
Press i
to switch to INSERT
mode, then find the line log 127.0.0.1 local2
and comment it out by adding a #
character to the beginning of the line, as highlighted in the following example:
. . .
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
#log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
. . .
Now add a line directly after the commented out line with the following contents:
log /dev/log local0
The entire section of /etc/haproxy/haproxy.cfg
that you edited should contain the following lines:
. . .
# local2.* /var/log/haproxy.log
#
#log 127.0.0.1 local2
log /dev/log local0
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
. . .
The chroot
line is important, because it restricts the HAProxy process to accessing files in the /var/lib/haproxy
directory only. The log /dev/log local0
line will create a file inside that directory that Rsyslog will use to collect log entries from.
Once you are finished editing the log
lines in /etc/haproxy/haproxy.cfg
, save and close the file by pressing ESC
, typing :wq
, and pressing ENTER
.
The last step that you need to complete in this section is to create the /var/lib/haproxy/dev
directory since it does not exist by default.
Create the directory using the mkdir
command and then restart HAProxy:
- sudo mkdir /var/lib/haproxy/dev
- sudo systemctl restart haproxy.service
You have now configured HAProxy to send its logs to a Unix domain socket that resides in /var/lib/haproxy/dev/log
. In the next step, you will configure Rsyslog to create and access the socket.
Rsyslog’s default configuration on CentOS 8 does not handle HAProxy logs. To collect logs from the HAProxy service, open a new file /etc/rsyslog.d/99-haproxy.conf
using vi
or your preferred editor:
- sudo vi /etc/rsyslog.d/99-haproxy.conf
Press i
to switch to INSERT
mode, then paste the following lines into the file:
$AddUnixListenSocket /var/lib/haproxy/dev/log
# Send HAProxy messages to a dedicated logfile
:programname, startswith, "haproxy" {
/var/log/haproxy.log
stop
}
The $AddUnixListenSocket
directive tells Rsyslog to create a Unix domain socket in the specified location, in this case /var/lib/haproxy/dev/log
. The :programname, startswith, "haproxy"
section specifies the file where Rsyslog will write the log entries to that it collects from the socket.
Once you are finished editing /etc/rsyslog.d/99-haproxy.conf
, save and close the file by pressing ESC
, typing :wq
, and pressing ENTER
.
You have now configured Rsyslog to read log entries from the Unix domain socket in /var/lib/haproxy/dev/log
and write them to a log file in /var/log/haproxy.log
.
However, before restarting Rsyslog you will need to determine if SELinux is enforcing access control on your CentOS 8 system.
To check SELinux’s current policy, run the following:
- getenforce
You will receive one of the following outputs:
Enforcing
- In this mode, SELinux is enforcing access controls on your system. You will need to complete the following optional Step 4 — Configuring SELinux section.Permissive
- In this case, SELinux logs all access attempts to its log file, but does not enforce access controls on your system.Disabled
- If SELinux is disabled, then it is not logging or enforcing any access control policies on your system.If the getenforce
command returned either Permissive
or Disabled
, then you can restart Rsyslog with the following command:
- sudo systemctl restart rsyslog
Once you restart Rsyslog, you will be able to view logs in the /var/log/haproxy.log
file that you configured in /etc/rsyslog.d/99-haproxy.conf
. Proceed to Step 5 — Testing HAProxy Logging to make sure that everything is working as expected.
Otherwise, if your system is running SELinux in Enforcing
mode, then the next section of this tutorial explains how to add a module to allow Rsyslog and HAProxy to communicate with each other over their shared Unix domain socket.
If your CentOS 8 system is configured with SELinux in Enforcing
mode, then you will need to allow Rsyslog access to HAProxy’s chroot
directory. Allowing this access will let Rsyslog create the Unix domain socket that HAproxy will send its logs to.
If you are not familiar with SELinux, this tutorial series An Introduction to SELinux on CentOS 7 will help you learn how to manage and interact with SELinux. Although it is written for CentOS 7, the principles and commands in the series are equally applicable to CentOS 8.
To enable Rsyslog and HAProxy access to their shared socket, the first task is to create a Type Enforcement policy file. Open a new file called rsyslog-haproxy.te
in vi
or your preferred editor:
- vi rsyslog-haproxy.te
Press i
to switch to INSERT
mode, then paste the following lines into the file:
module rsyslog-haproxy 1.0;
require {
type syslogd_t;
type haproxy_var_lib_t;
class dir { add_name remove_name search write };
class sock_file { create setattr unlink };
}
#============= syslogd_t ==============
allow syslogd_t haproxy_var_lib_t:dir { add_name remove_name search write };
allow syslogd_t haproxy_var_lib_t:sock_file { create setattr unlink };
The first line defines the module name and version. The require
portion tells the SELinux module loader about the types and classes that are required for the policy to be loaded as a module. The last two lines are the rules that allow Rsyslog access to HAProxy’s chroot
and socket file respectively.
When you are done editing the file, save and close it by pressing ESC
, typing :wq
, and pressing ENTER
.
Next, run the following command to install the checkpolicy
package, which contains the checkmodule
utility that you will use to turn the Type Enforcement file into an SELinux module.
- sudo dnf install checkpolicy
Now that you have the checkmodule
tool installed, the next step is to compile the module and then load it into SELinux. Run the following to compile the Type Enforcement file into an SELinux module:
- checkmodule -M -m rsyslog-haproxy.te -o rsyslog-haproxy.mod
Next, run semodule_package
to generate a complete policy package that SELinux can load into the Linux kernel:
- semodule_package -o rsyslog-haproxy.pp -m rsyslog-haproxy.mod
The final step is to load the package that you generated into the Linux kernel using the semodule
command:
- sudo semodule -i rsyslog-haproxy.pp
Adding the module may take a few seconds. Once the command completes you can confirm that the module is loaded into the kernel by running the semodule
command:
- sudo semodule -l |grep rsyslog-haproxy
You should receive output like the following:
Outputrsyslog-haproxy
Once the module is loaded you can restart Rsyslog with the following command:
- sudo systemctl restart rsyslog
You have now defined, compiled, and loaded an SELinux policy that will allow HAProxy and Rsyslog to communicate over their shared socket.
In the next step you will test that everything works by making an HTTP request to HAProxy and examining its new log file.
Now that you have configured HAProxy, Rsyslog, and optionally SELinux, you can test that logging to /var/log/haproxy.log
is working.
By default the haproxy
package ships with a configuration file that creates an HTTP listener socket on port 5000. The configuration points to a non-existent backend server, so any request to the port will result in an HTTP 503 error.
To check for a 503 error in your /var/log/haproxy.log
file, first generate an HTTP request using curl
like this:
- curl -si http://127.0.0.1:5000
You should receive output like the following:
OutputHTTP/1.0 503 Service Unavailable
Cache-Control: no-cache
Connection: close
Content-Type: text/html
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
Now examine /var/log/haproxy.log
for any HTTP 503 responses using the grep
command:
- sudo grep -E 'NOSRV.+503' /var/log/haproxy.log
Note: The NOSRV.+503
portion of the command is a regular expression. This tutorial on Using Grep & Regular Expressions to Search for Text Patterns in Linux
goes into more depth on using grep
and regular expressions.
You should receive a line (or multiple lines) like the following:
OutputSep 9 21:32:22 centos-s-1vcpu-1gb-nyc3-01 haproxy[4451]: 127.0.0.1:56024 [9/Sep/2020:21:32:22.098] main app/<NOSRV> 0/-1/-1/-1/0 503 212 - - SC-- 1/1/0/0/0 0/0 "GET / HTTP/1.1"
This line corresponds to the curl
request that you made, which means that Rsyslog and HAProxy are configured to use their shared socket correctly.
In this quickstart tutorial, you configured HAProxy to log to a Unix domain socket. You also set up Rsyslog to create and read from the socket so that the two programs can communicate with each other without opening up any TCP/IP ports on your system. Finally, you optionally compiled, packaged, and loaded an SELinux policy to allow Rsyslog and HAProxy shared access to their socket.
Thanks for learning with the DigitalOcean Community. Check out our offerings for compute, storage, networking, and managed databases.
This tutorial series explains how to troubleshoot and fix some of the most common errors that you may encounter when using the HAProxy TCP and HTTP proxy server.
Each tutorial in this series includes descriptions of common HAProxy configuration, network, filesystem, or permission errors. The series begins with an overview of the commands and log files that you can use to troubleshoot HAProxy. Subsequent tutorials examine specific errors in detail.
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
Sign up for Infrastructure as a Newsletter.
Working on improving health and education, reducing inequality, and spurring economic growth? We'd like to help.
Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.
I followed the steps to enable logging on CentOS 7, in /var/log/haproxy.log I’m seeing only startup logs and the Http logs. This is how my haproxy.cfg looks like -
global log /dev/log local0 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid stats timeout 30s user haproxy group haproxy daemon
defaults log global mode http option httplog option dontlognull timeout connect 5000 timeout client 50000 timeout server 50000
frontend app_front_end bind 0.0.0.0:8080 default_backend vmss_backend
backend vmss_backend balance roundrobin server server1 10.0.1.5:8080 check cookie server1 server server2 10.0.1.8:8080 check cookie server2
listen stats bind 0.0.0.0:9990 stats enable stats uri / stats realm Haproxy\ Statistics stats auth admin:admin