How can I get managed Kubernetes nodes with DO monitoring agent? I'd like to see RAM graphs...

November 2, 2018 2.9k views
Kubernetes Monitoring

I can't figure out a way to SSH on to those nodes (which makes sense); is there any way to get them provisioned with the monitoring agent enabled?

1 comment
  • I think the most k8s way, will be to add prometheus + grafana. It's easy to install you can use helm for this.

2 Answers

For others who come here from Google, this is the yaml I'm using to run the DO monitoring agent under DigitalOcean Kubernetes:

apiVersion: v1
kind: Namespace
metadata:
  name: do-agent
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: do-agent
  namespace: do-agent
spec:
  selector:
    matchLabels:
      name: do-agent
  template:
    metadata:
      labels:
        name: do-agent
    spec:
      volumes:
      - name: proc
        hostPath:
          path: /proc
      - name: sys
        hostPath:
          path: /proc
      containers:
      - name: do-agent
        image: digitalocean/do-agent:1.0.16-rc
        ports:
        - containerPort: 10253
          hostPort: 10253
          protocol: TCP
          name: metrics
        volumeMounts:
        - name: proc
          readOnly: true
          mountPath: /host/proc
        - name: sys
          readOnly: true
          mountPath: /host/sys
        imagePullPolicy: IfNotPresent
        securityContext:
          privileged: true
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      hostPID: true
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1

We're currently working on implementing Kubernetes monitoring in the DigitalOcean dashboard, including node performance metrics plus a handful of optional deployment-related kube-state-metrics. The do-agent will run on the nodes by default. We expect monitoring to be out by next month.

  • Are you planning to implement monitoring of load balancers created for kubernetes?

    • There is already monitoring for Kubernetes load balancers. If you click on the load balancer name for details you can access monitoring from the Graphs tab.

      • Thanks, I've found that out about 30 minutes ago! BUT: First of all I had to reconfigure my load balancers to use protocol of HTTP (not TCP) via annotations in manifest and route some serious traffic to it. (Hope that would be helpful for somebody)

        I guess there is serious rounding in it. Graphs didn't show anything with load about 1-2 req/sec (although there was some peaks up to 30 req/sec, I know that from my ingress-nginx stats).
        With 5-6 req/sec it started to work.

        This is very strange behavior. I understand that providing clear and highly detailed monitoring is expensive task, but is there any way to get it for DO's Load Balancers?

        • Hey there, PM for Load Balancers here.

          The short explanation is that the metrics poll on 5 minute intervals so it won't capture spikes during that period unless it's sustained throughout the 5 minutes, so you're really just getting an approximate.

          We haven't updated the metrics and graphs since we first launched the product and we hope to make improvements in the coming months, so this feedback is helpful.

Have another answer? Share your knowledge.