Enabling proxy protocol on a Kubernetes ingress load balancer only works with requests that come from outside the cluster. Requests from inside the cluster break.
This seems to be because requests from inside the cluster don’t get routed through the load balancer, so don’t get proxy protocol headers applied. The ingress controller then seems to fail to parse the request correctly, as it is expecting proxy protocol headers.
When proxy protocol is disabled, the ingress controller sees external traffic coming from a droplet subnet IP (presumably the loadbalancer IP), and internal traffic coming directly from a cluster pod subnet IP (interestingly, this isn’t the actual pod IP).
Note that this also means other DigitalOcean load balancer features don’t work for internal traffic, e.g. HTTPS redirection.
I haven’t been able to determine if this is due to the DigitalOcean LoadBalancer service implementation, or a feature of Kubernetes itself.
Either way, it means that I can’t use proxy protocol, as I have services that must be accessible inside and out of the cluster on the same hostname. It’d be great if there was some way to disable this behaviour, or some kind of workaround.
LoadBalancer service config for reference:
apiVersion: v1 kind: Service metadata: name: kong-proxy namespace: kong annotations: service.beta.kubernetes.io/do-loadbalancer-protocol: "http" service.beta.kubernetes.io/do-loadbalancer-algorithm: "round_robin" service.beta.kubernetes.io/do-loadbalancer-tls-ports: "443" service.beta.kubernetes.io/do-loadbalancer-tls-passthrough: "true" service.beta.kubernetes.io/do-loadbalancer-redirect-http-to-https: "true" service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true" spec: type: LoadBalancer selector: app: ingress-kong ports: - name: proxy port: 80 protocol: TCP targetPort: proxy - name: proxy-ssl port: 443 protocol: TCP targetPort: proxy-ssl externalTrafficPolicy: Local
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.
If anyone else has this issue, I eventually found it was caused by kube-proxy adding an iptables rule for the loadbalancer external IP, causing traffic to bypass the loadbalancer. There’s a kubernetes issue, and a workaround in the DigitalOcean examples.
Apologies for resurrecting this thread, but since it’s being linked to from elsewhere:
As an alternative, compumike created a drop-in workaround: hairpin-proxy.
Additionally, it should have been fixed in upstream by 1.20 https://github.com/kubernetes/kubernetes/pull/92312 but was reverted. The issue is still on the radar, however, so maybe 1.21.
Want to learn more? Join the DigitalOcean Community!
Join our DigitalOcean community of over a million developers for free! Get help and share knowledge in Q&A, subscribe to topics of interest, and get courses and tools that will help you grow as a developer and scale your project or business.
Sign up now