Question

How to expose tcp port of kubernetes-nginx-ingress in DigitalOcean managed Kubernetes

I am following the below the guide

https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/

I also have deployed the nginx-ingress in kube-system namespace . These are the yamls files applied

apiVersion: v1
kind: ConfigMap
metadata:
  name: tcp-nginx-rabbitmq
  namespace: kube-system
data:
  15672: "default/rabbitmq:15672"

apiVersion: v1
kind: Service
metadata:
  annotations:
    kompose.cmd: kompose convert --controller deployment -v -o DO-Kubernetes
    kompose.version: 1.21.0 (992df58d8)
  creationTimestamp: null
  labels:
    io.kompose.service: rabbitmq
  name: rabbitmq
spec:
  ports:
  - name: "15672"
    port: 15672
    targetPort: 15672
  - name: "5672"
    port: 5672
    targetPort: 5672
  selector:
    io.kompose.service: rabbitmq
status:
  loadBalancer: {}


apiVersion: v1
kind: Service
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protoco-nginx","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"ingress-nginx","app.kubernetes.io/version":"0.35.0","helm.sh/externalTrafficPolicy":"Local","ports":[{"name":"http","port":80,"protocol":"TCP","targetPort":"http"},{"name":"https","port":443,"prrnetes.io/instance":"ingress-nginx","app.kubernetes.io/name":"ingress-nginx"},"type":"LoadBalancer"}}
    kubernetes.digitalocean.com/load-balancer-id: <id>
    service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true"
  creationTimestamp: "2020-09-26T04:51:57Z"
  finalizers:
  - service.kubernetes.io/load-balancer-cleanup
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: 0.35.0
    helm.sh/chart: ingress-nginx-2.13.0
  name: ingress-nginx-controller
  namespace: kube-system
  resourceVersion: "11774380"
  selfLink: /api/v1/namespaces/kube-system/services/ingress-nginx-controller
  uid: <uid>
spec:
  clusterIP: 10.245.75.126
  externalTrafficPolicy: Local
  healthCheckNodePort: 30093
  ports:
  - name: http
    nodePort: 30583
    port: 80
    protocol: TCP
    targetPort: http
  - name: https
    nodePort: 30628
    port: 443
    protocol: TCP
    targetPort: https
  - name: rabbitmq-tcp-15672
    nodePort: 31555
    port: 15672
    protocol: TCP
    targetPort: 15672
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
- ip: <ip>

The service is up and I can see the starting logs of rabbitmq ….(no hits) . The tcp port is also open from the loadbalancer confirmed with nc -zv. Firefox is also giving me an error

Secure Connection failed 

ip:15672 PR_END_OF_FILE_ERROR 
Subscribe
Share

@hcgaron This config works for me:

controller:
  config:
    use-proxy-protocol: "true"

  service:
    annotations:
      service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true"

tcp:
  8883: "mqtt/mosquitto:8883:PROXY"

Word PROXY is crucial in this line:

8883: "mqtt/mosquitto:8883:PROXY"

@kennethafreelancer so am I . But it’s fun .

@kennethafreelancer

i am back again. Actually i got the https part working as well. in your case do a

kubectl edit svc ingress-nginx-controller -n ingress-nginx

and find the below lines

    spec:
      containers:
      - args:
        - /nginx-ingress-controller
        - --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
        - --election-id=ingress-controller-leader
        - --ingress-class=nginx
        - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
        - --validating-webhook=:8443
        - --validating-webhook-certificate=/usr/local/certificates/cert
        - --validating-webhook-key=/usr/local/certificates/key
        - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services


if u dont see --tcp-services-configmap add it like above.

cat nginx config

kubectl exec -it -n ingress-nginx deploy/ingress-nginx-controller -- cat /etc/nginx/nginx.conf  | grep <openvpn port> 

if u see some results then nginx knows about openvpn.

also discard my changes for removing port in configmap.

check nginx logs

kubectl logs -f --tail=10 deploy/ingress-nginx-controller  -n ingress-nginx

i removed the port in configmap and nginx logs said invalid entry.

Following the steps in this thread I got the tcp working, but it broke my http. Specifically setting the proxy protocol to false breaks my http.

On the other hand, setting it to “true” breaks tcp forwarding. Any advice?

@favas Thanks man, I am new to kubernetes and I almost give up for implement this untill I saw your last message.

@favas Brother thanks ALOT!!! I can confirm this solution worked for this version of DO deployment of ingress-nginx https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.40.2/deploy/static/provider/do/deploy.yaml, so to expose the tcp service and http, https app using the same loadbalancer is completely possible

@favas Thanks for reply, still no luck lol

@kennethafreelancer I see that proxy_protocol annotation is enabled in the service yaml of ingress-nginx. Can you change that to false .

service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: 'true'

So it’s two changes remove port 35088 and the proxy_protocol to false

@favas tried, still have the same issue, the port is opened but not routed to openvpn. I tried expose openvpn with loadBalance will work, the reason I want to use nginx ingress is I want to keep only one load balancer.

@kennethafreelancer

can you try removing 35088 .

9000: "default/openvpn2:35088"

Openvpn service yaml says the targetport is 443 but I am assuming 35088 in the configmap is the issue.


Submit an answer
You can type!ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

@pistle2020 How to expose multiple services using this tcp.

let say

apiVersion: v1 kind: ConfigMap metadata: name: tcp-services namespace: ingress-nginx-private data: 5432: “demo/postgres:5432”

this will expose “postgres” service. if we have 10 services which need to expose same like about how can we achieve this?

Ingress does not support TCP or UDP services. For this reason this Ingress controller uses the flags --tcp-services-configmap and --udp-services-configmap to point to an existing config map where the key is the external port to use and the value indicates the service to expose using the format: <namespace/service name>:<service port>:[PROXY]:[PROXY]

It is also possible to use a number or the name of the port. The two last fields are optional. Adding PROXY in either or both of the two last fields we can use Proxy Protocol decoding (listen) and/or encoding (proxy_pass) in a TCP service

The next example shows how to expose the service example-go running in the namespace default in the port 8080 using the port 9000

apiVersion: v1 kind: ConfigMap metadata: name: tcp-services namespace: ingress-nginx data: 9000: “default/example-go:8080” Since 1.9.13 NGINX provides UDP Load Balancing. The next example shows how to expose the service kube-dns running in the namespace kube-system in the port 53 using the port 53

apiVersion: v1 kind: ConfigMap metadata: name: udp-services namespace: ingress-nginx data: 53: “kube-system/kube-dns:53” If TCP/UDP proxy support is used, then those ports need to be exposed in the Service defined for the Ingress.

apiVersion: v1 kind: Service metadata: name: ingress-nginx namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx spec: type: LoadBalancer ports: - name: http port: 80 targetPort: 80 protocol: TCP - name: https port: 443 targetPort: 443 protocol: TCP - name: proxied-tcp-9000 port: 9000 targetPort: 9000 protocol: TCP selector: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx