How To Troubleshoot Common HAProxy Errors
This tutorial series explains how to troubleshoot and fix some of the most common errors that you may encounter when using the HAProxy TCP and HTTP proxy server.
Each tutorial in this series includes descriptions of common HAProxy configuration, network, filesystem, or permission errors. The series begins with an overview of the commands and log files that you can use to troubleshoot HAProxy. Subsequent tutorials examine specific errors in detail.
There are three main commands, and a common log location that you can use to get started troubleshooting HAProxy errors. Generally when you are troubleshooting HAProxy, you will use these commands in the order indicated here, and then examine the log file for specific diagnostic data.
The commands and log that you will commonly use to troubleshoot HAProxy across most Linux distributions are:
systemctl- Used to control and interact with Linux services via the
journalctl- Used to query and view the logs that are generated by
haproxy- When troubleshooting, this command is used to check HAProxy’s configuration.
/var/log/haproxy.log- This file contains log entries from HAProxy itself detailing TCP and HTTP traffic that is being handled by the server.
These commands, how to use them, and HAProxy’s logs where you can find additional information about errors are described in further detail in the following sections.
systemctl Commands for HAProxy
To troubleshoot common HAProxy errors using the
systemd service manager, the first step is to inspect the state of the HAProxy processes on your system. The following
systemctl commands will query
systemd for the state of HAProxy’s processes on most Linux distributions.
- sudo systemctl status haproxy.service -l --no-pager
-l flag will ensure that output is not truncated or ellipsized. The
--no-pager flag will make sure that output will go directly to your terminal without requiring any interaction on your part to view it. If you omit the
--no-pager flag you will be able to scroll through the output using arrow keys, or the page up and down keys. To quit from the pager use the
q key. You should receive output like this:
Output● haproxy.service - HAProxy Load Balancer Loaded: loaded (/lib/systemd/system/haproxy.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2020-08-20 19:30:11 UTC; 5s ago Docs: man:haproxy(1) file:/usr/share/doc/haproxy/configuration.txt.gz Process: 487 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS (code=exited, status=0/SUCCESS) Main PID: 488 (haproxy) Tasks: 2 (limit: 2344) . . . Aug 19 21:31:46 d6cdd0c71489 systemd: Started HAProxy Load Balancer.
Your output may be slightly different depending on which Linux distribution you are using, but in any case, make a note of the
Active line in the output. If your HAProxy server does not show
active (running) as highlighted in the example output but you expect it should, there may be an error. Typically if there is a problem, you will have a line like the following in your output (note the highlighted
Example Error OutputActive: failed (Result: exit-code) since Thu 2020-08-20 19:32:26 UTC; 6s ago
If there is a problem with your HAProxy process or configuration you can troubleshoot it further using the
journalctl Commands for HAProxy
To inspect the
systemd logs for HAProxy, you can use the
journalctl command. The
systemd logs for HAProxy will usually indicate whether there is a problem with starting or managing the HAProxy process.
These logs are separate from HAProxy’s request and error logs.
journalctl displays logs from
systemd that describe the HAProxy service itself, from startup to shutdown, along with any process errors that may be encountered along the way.
- sudo journalctl -u haproxy.service --since today --no-pager
--since today flag will limit the output of the command to log entries beginning at 00:00:00 of the current day only. Using this option will help restrict the volume of log entries that you need to examine when checking for errors. You should receive output like the following (there may be a few extra lines between the
Started lines depending on your Linux distribution):
OutputAug 20 19:37:08 d6cdd0c71489 systemd: Starting HAProxy Load Balancer... . . . Aug 20 19:37:08 d6cdd0c71489 systemd: Started HAProxy Load Balancer.
If there is an error, you will have a line in the output that is similar to the following, with the main difference between Linux distributions being the highlighted
Example Error OutputAug 20 19:32:25 yourhostname systemd: Failed to start HAProxy Load Balancer.
If your HAProxy server has errors in the
journalctl logs like the previous example, then the next step to troubleshoot possible issues is investigating HAProxy’s configuration using the
haproxy command line tool.
To troubleshoot HAProxy configuration issues, use the
haproxy -c command. The tool will parse your HAProxy files and detect any errors or missing settings before attempting to start the server.
Run the command like this on Ubuntu, Debian, CentOS, and Fedora based distributions. Be sure to change the path to the configuration file if you are using a different filename or location:
- sudo haproxy -c -f /etc/haproxy/haproxy.cfg
A working HAProxy configuration will result in output like the following:
OutputConfiguration file is valid
If there is an error in your HAProxy configuration, like a typo or misplaced directive,
haproxy -c will detect it and attempt to notify you about the problem.
For example, attempting to use the
bind directive in
haproxy.cfg in the wrong location will result in messages like the following:
Example Error Output[ALERT] 232/194354 (199) : parsing [/etc/haproxy/haproxy.cfg:13] : unknown keyword 'bind' in 'global' section [ALERT] 232/194354 (199) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg [ALERT] 232/194354 (199) : Fatal errors found in configuration.
In this example the
bind directive is misplaced inside a
global configuration section, so HAProxy generates the
unknown keyword error. The message also includes a line number
13, so that you can edit the file and fix or remove the erroneous line without having to search through the file.
Learning how to use
haproxy -c to detect and fix errors is useful when you are troubleshooting an existing error, or before you reload HAProxy with an edited configuration that may contain errors.
HAProxy Log Files
HAProxy log files are a very helpful resource for troubleshooting. Generally, any error that you receive in a browser or other HTTP client will have a corresponding entry in HAProxy’s logs. Sometimes HAProxy will also output errors related to configuration and other debugging information to its log files.
On Ubuntu and Debian based Linux distributions, the
haproxy package includes scripts that configure log output in
On CentOS, Fedora, and other RedHat-derived Linux distributions,
haproxy does not output to a log file by default. To log HAProxy output logs to
/var/log/haproxy.log, follow this quickstart tutorial, How To Configure HAProxy Logging with Rsyslog on CentOS 8.
When you are troubleshooting HAProxy using its log file, examine
/var/log/haproxy.log for errors using a tool like
less. For example, to view the last two lines of the log using
tail, run the following command:
- sudo tail -n 2 /var/log/haproxy.log
An example error will resemble something like the following lines, regardless of which Linux distribution you are using to run your HAProxy server:
Log ExamplesAug 20 19:36:21 d6cdd0c71489 haproxy: [ALERT] 258/134605 (19202) : Proxy 'app', server 'app1' [/etc/haproxy/haproxy.cfg:88] verify is enabled by default but no CA file specified. If you're running on a LAN where you're certain to trust the server's certificate, please set an explicit 'verify none' statement on the 'server' line, or use 'ssl-server-verify none' in the global section to disable server-side verifications by default. Aug 20 19:36:22 d6cdd0c71489 haproxy: 203.0.113.1:54428 [20/Aug/2020:19:36:22.288] main app/<NOSRV> 0/-1/-1/-1/1 503 212 - - SC-- 1/1/0/0/0 0/0 "GET / HTTP/1.1"
These example lines are just for illustration purposes. If you are diagnosing errors with your HAProxy server, chances are the lines in your logs will have different contents than these. Some lines will include success responses and other non-critical diagnostic entries.
Regardless of your Linux distribution, the format of the lines in your HAProxy logs will include any HTTP status codes that are returned to clients, along with requesting IPs and the status of backend servers.
Once you have an idea of what might be causing problems with your HAProxy server you can continue researching and troubleshooting the issue. The HTTP status code and text description are especially useful, since they give you explicit and specific terms that you can use to narrow down the range of possible causes of a problem.
Troubleshooting HAProxy errors can range from diagnosing errors with the service itself, to locating misconfigured options for modules, or to examining customized access control rules in detail. This introduction to diagnosing issues with HAProxy explained how to use a number of utilities to help narrow down the possible causes of errors. Usually, you will use these utilities in the same order, although you can always skip some, or start directly with examining logs if you have a general idea of what the problem might be.
However, as a general sequence for troubleshooting, it helps to be methodical and use these tools in the order described. Start troubleshooting with
systemctl to examine the state of the HAProxy server. If you need more information, examine the
systemd logs for HAProxy using the
journalctl command. If the issue is still not apparent after checking
journalctl, testing HAProxy’s configuration using
haproxy -c -f /etc/haproxy/haproxy.cfg is the next step. Finally, for in-depth troubleshooting, examining HAProxy’s log files will usually indicate a specific error, with helpful diagnostic messages and error codes.
The rest of the tutorials in this series will examine some common errors that you may encounter when using HAProxy in more detail.