Question

How can I get Socket.io to work on Kubernetes when using nginx-ingress and cert-manager?

Posted February 25, 2020 2.4k views
NginxLoad BalancingKubernetes

I have a deployment running inside my k8s cluster that uses Socket.io, but external WebSocket traffic doesn’t seem to be getting to it.

I’m using nginx ingress controller with cert-manager, which works fine for normal HTTPS traffic. With this setup, SSL termination is with nginx and the certificates live in the cluster.

I’ve seen in the docs and elsewhere that I need to switch the load balancer protocol to HTTP instead of TCP to get WebSockets to work. However, doing that breaks SSL for me. This is because the DigitalOcean load balancer has two options:

  1. SSL Termination at the load balancer - this would require managing my domain and SSL certs through DigitalOcean and abandoning cert-manager which I don’t want to do
  2. SSL Passthrough - this passes the encrypted data straight to the droplet, which is also not what I want to do (I think??)

Is there a way to continue to use nginx ingress controller and cert manager with WebSockets?

1 comment
  • I am likewise having serious Socket.io/Websockets issues. I can’t find any official documentation on how to deal with this scenario, but it seems like a pretty common situation. I know that “me too!” comments aren’t overly helpful, but I’m hoping that more activity may garner a response from DO.

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
2 answers

Did you ever get a resolution to this, I am stuck with exactly the same issue.

I got this working in the end, terminating the SSL at nginx (passthrough on the load balancer) and allowing it to reverse-proxy the data to the apps with sticky sessions. For anyone else stuck on this, here are my manifests - hope they help:

nginx-ingress-loadbalancer-service.yaml

kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  annotations: 
    # See https://github.com/digitalocean/digitalocean-cloud-controller-manager/blob/master/docs/controllers/services/examples/README.md#accessing-pods-over-a-managed-load-balancer-from-inside-the-cluster
    service.beta.kubernetes.io/do-loadbalancer-name: "api.example.com"
    service.beta.kubernetes.io/do-loadbalancer-hostname: "api.example.com"
    service.beta.kubernetes.io/do-loadbalancer-protocol: "http"
    service.beta.kubernetes.io/do-loadbalancer-tls-passthrough: "true"
    service.beta.kubernetes.io/do-loadbalancer-sticky-sessions-type: "cookies"
    service.beta.kubernetes.io/do-loadbalancer-sticky-sessions-cookie-name: "jammydodger"
    service.beta.kubernetes.io/do-loadbalancer-sticky-sessions-cookie-ttl: "60"
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  externalTrafficPolicy: Local
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
    - name: https
      port: 443
      protocol: TCP
      targetPort: https

---

ingress.yaml

# Define Ingress for the service
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: testing-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/rewrite-target: /$2 # Lets us rewrite example.com/something/hello to example.com/hello
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/session-cookie-name: "route"
    nginx.ingress.kubernetes.io/session-cookie-hash: "sha1"
    nginx.ingress.kubernetes.io/session-cookie-expires: "172800"
    nginx.ingress.kubernetes.io/session-cookie-max-age: "172800"

spec:
  tls:
  - hosts:
    - api.example.com
    - example.com
    secretName: testing-ingress-tls
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /v0(/|$)(.*)
        backend:
          serviceName: platform-app-service
          servicePort: 80
  - host: example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: frontend-app-service
          servicePort: 80
Submit an Answer