Get metrics from Kubernetes nodes

October 21, 2018 2.5k views
Kubernetes

I would like to get advanced metrics from my Kubernetes worker node.

When I view the worker nodes in the Droplets screen, I get the usual instruction to shell into the Droplet and install the DO Agent, but I cannot shell into the worker.

Step 3 of the Kubernetes Quick Start guide says to choose a data center and an SSH key, but I do not have an SSH key option when I create a cluster so I appear to be unable to shell in to my worker node and install the DO Agent.

Is it possible to install the DO Agent on worker nodes?

Thanks!

5 Answers
Trurl October 26, 2018
Accepted Answer

I'm use my own wrapper for it

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: do-agent
  namespace: monitoring
spec:
  selector:
    matchLabels:
      name: do-agent
      phase: prod
  template:
    metadata:
      labels:
        name: do-agent
        phase: prod
#      annotations:
#        prometheus.io/port: "10253"
#        prometheus.io/path: "/metrics"
#        prometheus.io/scrape: "true"
    spec:
      volumes:
      - name: proc
        hostPath:
          path: /proc
      containers:
      - name: do-agent
        image: trurlmcbyte/do-agent:0.5.1
#        command: ["/agent/do-agent_linux_amd64", "-log_level", "DEBUG"]
        ports:
        - containerPort: 10253
          hostPort: 10253
          protocol: TCP
          name: metrics
        volumeMounts:
        - name: proc
          readOnly: true
          mountPath: /agent/proc
        imagePullPolicy: IfNotPresent
        securityContext:
          privileged: true
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      hostPID: true
      schedulerName: default-scheduler
      tolerations:
      - operator: Exists
        effect: NoSchedule
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  templateGeneration: 1
  revisionHistoryLimit: 10

wrapper itself is simple:

FROM golang:1.9-alpine

ENV CGO=0
ENV GOOS=linux

ARG CURRENT_BRANCH
ARG CURRENT_HASH
ARG LAST_RELEASE

RUN  apk update && \
     apk add bash && \
     apk add curl && \
     apk add git && \
     apk add make && \
     apk add libc6-compat

COPY ./do-agent /go/src/github.com/digitalocean/do-agent

RUN cd /go/src/github.com/digitalocean/do-agent && \
    set -x && \
    make build RELEASE=${LAST_RELEASE} CURRENT_BRANCH=${CURRENT_BRANCH} CURRENT_HASH=${CURRENT_HASH}

# Copy what is needed to
FROM alpine
ENV DO_AGENT_REPO_PATH   /agent/updates
ENV DO_AGENT_PROCFS_ROOT /agent/proc

RUN mkdir -p /agent
RUN mkdir -p /agent/updates
RUN mkdir -p /agent/proc

RUN  apk update && \
     apk add libc6-compat && \
     apk add ca-certificates

COPY --from=0 /go/src/github.com/digitalocean/do-agent/build/do-agent_linux_amd64 /agent
RUN find /agent

CMD /agent/do-agent_linux_amd64
  • Interesting! It seems a shame to have to use Kubernetes to do that but until I can shell in to my worker droplets this will do. Thanks!

    • Workaround to ssh into a worker: Reset the root password

      1. Go to Droplets > Select droplet worker > Access > Reset root password
      2. Check email for the new password
      3. Access the console from the DO droplet web dashboard (SSH access is not allowed by DO's cloud firewall, which is good)
      4. Enter "root" as the user and the password from the email. You'll be forced to change the password right after a successful login.

      I hope this helps.

      • Ah interesting, I'd wondered about doing that but had been worried it would break things. Thanks!

        • It shouldn't break things but requires a restart of your droplet, so please don't do it if you only have one worker in your cluster and you don't want to have some downtime.
          Cheers!

Kubernetes monitoring is in progress, and should be out by next month. This will provide worker performance metrics in the DigitalOcean dashboard, as well as a handful of optional deployment-related kube-state-metrics. The do-agent will be installed by default.

I have the same problem, I just can connect to the kubernetes cluster itself

Updated daemonset.yaml to use DO's official docker image:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: do-agent
  namespace: default
spec:
  selector:
    matchLabels:
      name: do-agent
      phase: prod
  template:
    metadata:
      labels:
        name: do-agent
        phase: prod
    spec:
      volumes:
      - name: proc
        hostPath:
          path: /proc
      - name: sys
        hostPath:
          path: /sys
      containers:
      - name: do-agent
        image: digitalocean/do-agent:1.1.3-rc
        command: ["/bin/do-agent", "--path.procfs", "/host/proc", "--path.sysfs", "/host/sys"]
        ports:
        - containerPort: 10253
          hostPort: 10253
          protocol: TCP
          name: metrics
        volumeMounts:
        - name: proc
          readOnly: true
          mountPath: /host/proc
        - name: sys
          readOnly: true
          mountPath: /host/sys
        imagePullPolicy: IfNotPresent
        securityContext:
          privileged: true
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      hostPID: true
      schedulerName: default-scheduler
      tolerations:
      - operator: Exists
        effect: NoSchedule
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  templateGeneration: 1
  revisionHistoryLimit: 10
Have another answer? Share your knowledge.