HAProxy failing at random times (backend has no server available!)
I have a CentOS 7.1 (x64) droplet configured to load balance between 2 apache/php servers.
I have it working perfectly, but randomly throughout the day it reports the following:
haproxy: backend app_backend has no server available!
HA Proxy Log File:
Nov 5 : Server app_backend/web-live is DOWN, reason: Layer4 timeout, check duration: 2001ms. 0 active and 1 backup servers left. Running on backup. 11 sessions active, 0 requeued, 0 remaining in queue. Nov 5: Backup Server app_backend/web-backup is DOWN, reason: Layer4 timeout, check duration: 2001ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. Nov 5: backend app_backend has no server available!
When visiting the front end it shows a 503 error, then within a few seconds it all comes back up again.
My HA Proxy Config:
global log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon stats socket /var/lib/haproxy/stats defaults mode http log global option httplog option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout http-keep-alive 10s timeout check 10s maxconn 3000 frontend tm_application bind *:80 mode http option forwardfor default_backend app_backend backend app_backend mode http balance roundrobin option forwardfor option httpchk server web-live xx.xx.xx.xx:80 check server web-backup xx.xx.xx.xx:80 check backup
Can anyone point me at what could be wrong? to me it seems that it is trying to check 2000ms and due to server response it might be failing?