Related

Product Managed Load Balancers on DigitalOcean Product
Using PROXY protocol for tcp-services in Kubernetes Question Question
413 Request Entity Too Large - NGINX Question Question

Question

How can I get Socket.io to work on Kubernetes when using nginx-ingress and cert-manager?

Posted February 25, 2020 9.3k views
NginxLoad BalancingKubernetes

I have a deployment running inside my k8s cluster that uses Socket.io, but external WebSocket traffic doesn’t seem to be getting to it.

I’m using nginx ingress controller with cert-manager, which works fine for normal HTTPS traffic. With this setup, SSL termination is with nginx and the certificates live in the cluster.

I’ve seen in the docs and elsewhere that I need to switch the load balancer protocol to HTTP instead of TCP to get WebSockets to work. However, doing that breaks SSL for me. This is because the DigitalOcean load balancer has two options:

  1. SSL Termination at the load balancer - this would require managing my domain and SSL certs through DigitalOcean and abandoning cert-manager which I don’t want to do
  2. SSL Passthrough - this passes the encrypted data straight to the droplet, which is also not what I want to do (I think??)

Is there a way to continue to use nginx ingress controller and cert manager with WebSockets?

Add a comment
Huggzorx
By:
Huggzorx

Related

Product Managed Load Balancers on DigitalOcean Product
Using PROXY protocol for tcp-services in Kubernetes Question Question
413 Request Entity Too Large - NGINX Question Question
Add a comment
1 comment
  • amartincolby May 11, 2020
    Reply Report

    I am likewise having serious Socket.io/Websockets issues. I can’t find any official documentation on how to deal with this scenario, but it seems like a pretty common situation. I know that “me too!” comments aren’t overly helpful, but I’m hoping that more activity may garner a response from DO.

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
Submit an Answer
4 answers
RobinJ July 1, 2020
[deleted]
Reply Report
mattbarnden September 10, 2020

Did you ever get a resolution to this, I am stuck with exactly the same issue.

Reply Report
Huggzorx September 10, 2020

I got this working in the end, terminating the SSL at nginx (passthrough on the load balancer) and allowing it to reverse-proxy the data to the apps with sticky sessions. For anyone else stuck on this, here are my manifests - hope they help:

nginx-ingress-loadbalancer-service.yaml

kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  annotations: 
    # See https://github.com/digitalocean/digitalocean-cloud-controller-manager/blob/master/docs/controllers/services/examples/README.md#accessing-pods-over-a-managed-load-balancer-from-inside-the-cluster
    service.beta.kubernetes.io/do-loadbalancer-name: "api.example.com"
    service.beta.kubernetes.io/do-loadbalancer-hostname: "api.example.com"
    service.beta.kubernetes.io/do-loadbalancer-protocol: "http"
    service.beta.kubernetes.io/do-loadbalancer-tls-passthrough: "true"
    service.beta.kubernetes.io/do-loadbalancer-sticky-sessions-type: "cookies"
    service.beta.kubernetes.io/do-loadbalancer-sticky-sessions-cookie-name: "jammydodger"
    service.beta.kubernetes.io/do-loadbalancer-sticky-sessions-cookie-ttl: "60"
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  externalTrafficPolicy: Local
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
    - name: https
      port: 443
      protocol: TCP
      targetPort: https

---

ingress.yaml

# Define Ingress for the service
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: testing-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/rewrite-target: /$2 # Lets us rewrite example.com/something/hello to example.com/hello
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/session-cookie-name: "route"
    nginx.ingress.kubernetes.io/session-cookie-hash: "sha1"
    nginx.ingress.kubernetes.io/session-cookie-expires: "172800"
    nginx.ingress.kubernetes.io/session-cookie-max-age: "172800"

spec:
  tls:
  - hosts:
    - api.example.com
    - example.com
    secretName: testing-ingress-tls
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /v0(/|$)(.*)
        backend:
          serviceName: platform-app-service
          servicePort: 80
  - host: example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: frontend-app-service
          servicePort: 80
Reply Report
jkwiatkoski October 12, 2020

Looks like a user put a nice write-up for this over on Medium that you may find helpful:

https://medium.com/swlh/how-to-use-web-sockets-socket-io-with-digital-ocean-load-balancers-and-kubernetes-dok8s-with-e4dd5531c67e

Reply Report

Related

Looking for something else?

Ask a new question Search for more help