How to support internal traffic with proxy protocol enabled on a Kubernetes loadbalancer

Enabling proxy protocol on a Kubernetes ingress load balancer only works with requests that come from outside the cluster. Requests from inside the cluster break.

This seems to be because requests from inside the cluster don’t get routed through the load balancer, so don’t get proxy protocol headers applied. The ingress controller then seems to fail to parse the request correctly, as it is expecting proxy protocol headers.

When proxy protocol is disabled, the ingress controller sees external traffic coming from a droplet subnet IP (presumably the loadbalancer IP), and internal traffic coming directly from a cluster pod subnet IP (interestingly, this isn’t the actual pod IP).

Note that this also means other DigitalOcean load balancer features don’t work for internal traffic, e.g. HTTPS redirection.

I haven’t been able to determine if this is due to the DigitalOcean LoadBalancer service implementation, or a feature of Kubernetes itself.

Either way, it means that I can’t use proxy protocol, as I have services that must be accessible inside and out of the cluster on the same hostname. It’d be great if there was some way to disable this behaviour, or some kind of workaround.

LoadBalancer service config for reference:

apiVersion: v1
kind: Service
  name: kong-proxy
  namespace: kong
  annotations: "http" "round_robin" "443" "true" "true" "true"
  type: LoadBalancer
    app: ingress-kong
  - name: proxy
    port: 80
    protocol: TCP
    targetPort: proxy
  - name: proxy-ssl
    port: 443
    protocol: TCP
    targetPort: proxy-ssl
  externalTrafficPolicy: Local

Submit an answer
Answer a question...

This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

Sign In or Sign Up to Answer

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

Accepted Answer

If anyone else has this issue, I eventually found it was caused by kube-proxy adding an iptables rule for the loadbalancer external IP, causing traffic to bypass the loadbalancer. There’s a kubernetes issue, and a workaround in the DigitalOcean examples.

Apologies for resurrecting this thread, but since it’s being linked to from elsewhere:

As an alternative, compumike created a drop-in workaround: hairpin-proxy.

Additionally, it should have been fixed in upstream by 1.20 but was reverted. The issue is still on the radar, however, so maybe 1.21.

Want to learn more? Join the DigitalOcean Community!

Join our DigitalOcean community of over a million developers for free! Get help and share knowledge in Q&A, subscribe to topics of interest, and get courses and tools that will help you grow as a developer and scale your project or business.