How to Safely Configure Cilium on DOKS

Question

I wanted to try DOKS due to the competitive and transparent pricing model. DOKS ships with Cilium enabled as the default CNI. Cilium is awesome, however I noticed the DOKS-enabled Cilium offering is not very configurable nor flexible from a user perspective. In other clusters (a mix of both self-managed and managed), I am using a custom-installed Cilium with certain features enabled, for example:

Cilium’s integrated ingress controller
Cilium’s sidecarless service mesh
Cilium’s native support for completely replacing kube-proxy (i.e. removal of kube-proxy)

Does anyone have any patterns for safely configuring Cilium on DOKS, or should I look elsewhere for a managed Kubernetes offering? My proposition is allowing the DOKS bootstrapping to be more configurable, such as having an advanced option of not enabling a CNI at all so that it is up to the user for those that want to configure their own (Cilium, Calico, …).

This is possible on AWS EKS, Azure AKS, and GCP GKE, so would be a huge value prop for DigitalOcean imo.

Brian · Accepted Answer

I ended up trialing Civo , which supports safely “overlaying” their baked-in Cilium with a custom-installed Cilium Helm chart. When configuring a Civo cluster, there is the option for Flannel or Cilium CNI plugins. Select Cilium, install the Cilium Helm chart (be sure to pass --reuse-values to avoid overwriting the Civo Cilium configuration), and then you should be good to go. You can use any Cilium features after doing that as far as I’m aware. I was unable to do this on DOKS (overlay the default Cilium configuration with a custom Cilium Helm chart), but perhaps I missed a step. Please let me know if this is possible, and I will update this solution accordingly (as that would solve my question). I will keep an eye on DOKS support for safely configuring Cilium, but this is a good workaround if you are flexible to cloud providers and don’t mind using k3s (managed). See here for a similar guide: https://blog.ediri.io/how-to-enable-the-cilium-hubble-ui-in-a-civo-k3s-cluster#heading-enable-hubble-ui

jon richter · Answer

I believe it is safe to replace the managed Cilium with a self-managed repository. In our case, we are using ArgoCD for managing application manifests. We have good results with replacing the original DOKS Cilium with:

project: default
source:
  repoURL: 'https://helm.cilium.io/'
  targetRevision: 1.10.15
  helm:
    parameters:
      - name: k8sServiceHost
        value: <UUID>.k8s.ondigitalocean.com
      - name: kubeProxyReplacement
        value: partial
      - name: wellKnownIdentities.enabled
        value: 'true'
      - name: nodePort.enableHealthCheck
        value: 'false'
      - name: ipam.operator.clusterPoolIPv4PodCIDR
        value: 10.0.0.0/16
      - name: ipam.operator.clusterPoolIPv4MaskSize
        value: '25'
      - name: l2NeighDiscovery.refreshPeriod
        value: 30s
      - name: nativeRoutingCIDR
        value: 10.0.0.0/16
      - name: autoDirectNodeRoutes
        value: 'true'
      - name: tunnel
        value: disabled
      - name: operator.replicas
        value: '1'
      - name: operator.updateStrategy.rollingUpdate.maxSurge
        value: '0'
      - name: updateStrategy.rollingUpdate.maxUnavailable
        value: 10%
  chart: cilium
destination:
  server: 'https://kubernetes.default.svc'
  namespace: kube-system
syncPolicy: {}

These settings would be applicable to any other Helm based Cilium deployment.

This way we imitate the original configuration, where ArgoCD’s diff feature will be of great help, and then incrementally update.

Please make sure to follow the DOKS Changelog in so you can mimic the original DOKS settings in your derived setup:

https://docs.digitalocean.com/products/kubernetes/details/changelog/

jon richter · Answer

A similar, more specific request about strict kube-proxy replacement, has previously been filed into: https://ideas.digitalocean.com/app-framework-services/p/implement-strict-mode-for-cilium

Question

How to Safely Configure Cilium on DOKS

Try DigitalOcean for free

Popular Topics

Get our biweekly newsletter

Hollie's Hub for Good

Become a contributor

Featured on Community

DigitalOcean Products

Welcome to the developer cloud