Question

How to support internal traffic with proxy protocol enabled on a Kubernetes loadbalancer

Enabling proxy protocol on a Kubernetes ingress load balancer only works with requests that come from outside the cluster. Requests from inside the cluster break.

This seems to be because requests from inside the cluster don’t get routed through the load balancer, so don’t get proxy protocol headers applied. The ingress controller then seems to fail to parse the request correctly, as it is expecting proxy protocol headers.

When proxy protocol is disabled, the ingress controller sees external traffic coming from a droplet subnet IP (presumably the loadbalancer IP), and internal traffic coming directly from a cluster pod subnet IP (interestingly, this isn’t the actual pod IP).

Note that this also means other DigitalOcean load balancer features don’t work for internal traffic, e.g. HTTPS redirection.

I haven’t been able to determine if this is due to the DigitalOcean LoadBalancer service implementation, or a feature of Kubernetes itself.

Either way, it means that I can’t use proxy protocol, as I have services that must be accessible inside and out of the cluster on the same hostname. It’d be great if there was some way to disable this behaviour, or some kind of workaround.

LoadBalancer service config for reference:

apiVersion: v1
kind: Service
metadata:
  name: kong-proxy
  namespace: kong
  annotations:
    service.beta.kubernetes.io/do-loadbalancer-protocol: "http"
    service.beta.kubernetes.io/do-loadbalancer-algorithm: "round_robin"
    service.beta.kubernetes.io/do-loadbalancer-tls-ports: "443"
    service.beta.kubernetes.io/do-loadbalancer-tls-passthrough: "true"
    service.beta.kubernetes.io/do-loadbalancer-redirect-http-to-https: "true"
    service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true"
spec:
  type: LoadBalancer
  selector:
    app: ingress-kong
  ports:
  - name: proxy
    port: 80
    protocol: TCP
    targetPort: proxy
  - name: proxy-ssl
    port: 443
    protocol: TCP
    targetPort: proxy-ssl
  externalTrafficPolicy: Local

Submit an answer


This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

Sign In or Sign Up to Answer

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

Accepted Answer

If anyone else has this issue, I eventually found it was caused by kube-proxy adding an iptables rule for the loadbalancer external IP, causing traffic to bypass the loadbalancer. There’s a kubernetes issue, and a workaround in the DigitalOcean examples.

Apologies for resurrecting this thread, but since it’s being linked to from elsewhere:

As an alternative, compumike created a drop-in workaround: hairpin-proxy.

Additionally, it should have been fixed in upstream by 1.20 https://github.com/kubernetes/kubernetes/pull/92312 but was reverted. The issue is still on the radar, however, so maybe 1.21.

Try DigitalOcean for free

Click below to sign up and get $200 of credit to try our products over 60 days!

Sign up

Get our biweekly newsletter

Sign up for Infrastructure as a Newsletter.

Hollie's Hub for Good

Working on improving health and education, reducing inequality, and spurring economic growth? We'd like to help.

Become a contributor

Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.

Welcome to the developer cloud

DigitalOcean makes it simple to launch in the cloud and scale up as you grow — whether you're running one virtual machine or ten thousand.

Learn more
DigitalOcean Cloud Control Panel