Using PROXY protocol for tcp-services in Kubernetes

January 29, 2019 1.8k views
Kubernetes Nginx Load Balancing

I have a Kubernetes cluster that uses nginx for handling inbound requests behind a DigitalOcean load balancer. Here’s what an inbound SMTP request (port 25) goes through to get to my Postfix server:

  1. DO Load Balancer
  2. Nginx k8s service/loadbalancer
  3. Nginx server
  4. Postfix k8s service
  5. Postfix pod

It works fine, but for one problem: the originating IP of the client is lost along the way, and gets replaced with the IP address of the nginx server (10.x.y.z). This means that things like SPF policy checks don’t work, because Postfix cannot determine the IP of the sender.

There’s a way to fix it though, and that’s to have nginx use the PROXY protocol when communicating with Postfix. To do that, one should configure Postfix to expect the PROXY protocol for inbound requests (easy), and tell nginx to use the Proxy protocol when communicating with Postfix. That hasn’t been easy.

Per the Documentation, I should configure nginx’s tcp-services ConfigMap with a line like this (I think): 30025: mail/postfix:25, where 30025 is the NodePort that was assigned to the nginx load balancer, and 25 is the port that the Postfix k8s service is listening on.

When I configure it this way it correctly routes mail, but I’m still getting 10.x.y.z IP addresses in Postfix. Is this because the DigitalOcean load balancer isn’t using the PROXY protocol when communicating with nginx, and Postfix is seeing the internal IP address of the DO load balancer? Is there any way to configure the DO loadbalancer to use the PROXY protocol for non-HTTP services?

3 Answers

Support for PROXY protocol – including integration with DOKS – is available as of yesterday: https://blog.digitalocean.com/load-balancers-now-support-proxy-protocol/

Hey friend,

Our load balancers should pass the client IP as X-Forwarded-For. Here’s some discussion on pulling that with Nginx:

https://stackoverflow.com/questions/25929599/nginx-replace-remote-addr-with-x-forwarded-for

Some more local conversation here:

https://www.digitalocean.com/community/questions/using-digitalocean-loadbalancer-how-to-get-real-ip

Jarland

  • SMTP doesn’t have the notion of X-Forwarded-For; that’s an HTTP header.

  • So again, the question is, can I configure the DO Load Balancer to do a “TCP” forward using PROXY? It sounds like the answer is, “no.”

Have another answer? Share your knowledge.