I can't figure out a way to SSH on to those nodes (which makes sense); is there any way to get them provisioned with the monitoring agent enabled?
For others who come here from Google, this is the yaml I'm using to run the DO monitoring agent under DigitalOcean Kubernetes:
apiVersion: v1
kind: Namespace
metadata:
name: do-agent
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: do-agent
namespace: do-agent
spec:
selector:
matchLabels:
name: do-agent
template:
metadata:
labels:
name: do-agent
spec:
volumes:
- name: proc
hostPath:
path: /proc
- name: sys
hostPath:
path: /proc
containers:
- name: do-agent
image: digitalocean/do-agent:1.0.16-rc
ports:
- containerPort: 10253
hostPort: 10253
protocol: TCP
name: metrics
volumeMounts:
- name: proc
readOnly: true
mountPath: /host/proc
- name: sys
readOnly: true
mountPath: /host/sys
imagePullPolicy: IfNotPresent
securityContext:
privileged: true
restartPolicy: Always
terminationGracePeriodSeconds: 30
hostNetwork: true
hostPID: true
updateStrategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
Where did you get the image?
digitalocean/do-agent:1.0.16-rc
Edit:
Found https://hub.docker.com/r/digitalocean/do-agent/tags
changed digitalocean/do-agent:1.0.16-rc to digitalocean/do-agent:1.0.15-rc
And it's working now
We're currently working on implementing Kubernetes monitoring in the DigitalOcean dashboard, including node performance metrics plus a handful of optional deployment-related kube-state-metrics. The do-agent will run on the nodes by default. We expect monitoring to be out by next month.
Are you planning to implement monitoring of load balancers created for kubernetes?
There is already monitoring for Kubernetes load balancers. If you click on the load balancer name for details you can access monitoring from the Graphs tab.
Thanks, I've found that out about 30 minutes ago! BUT: First of all I had to reconfigure my load balancers to use protocol of HTTP (not TCP) via annotations in manifest and route some serious traffic to it. (Hope that would be helpful for somebody)
I guess there is serious rounding in it. Graphs didn't show anything with load about 1-2 req/sec (although there was some peaks up to 30 req/sec, I know that from my ingress-nginx stats).
With 5-6 req/sec it started to work.
This is very strange behavior. I understand that providing clear and highly detailed monitoring is expensive task, but is there any way to get it for DO's Load Balancers?
Hey there, PM for Load Balancers here.
The short explanation is that the metrics poll on 5 minute intervals so it won't capture spikes during that period unless it's sustained throughout the 5 minutes, so you're really just getting an approximate.
We haven't updated the metrics and graphs since we first launched the product and we hope to make improvements in the coming months, so this feedback is helpful.
I think the most k8s way, will be to add prometheus + grafana. It's easy to install you can use helm for this.