Using DO managed kubernetes cluster with helm chart stable/prometheus results in some node_exporters being unreachable.
I have three nodes in the cluster. Prometheus pods (which include server, alertmanager, nodeexporter etc) start just fine. Unfortunately 2 of the 3 nodeexporters cannot be reached. This seems like it must be some issue with flannel, but I don't know how to begin to debug this.
Prometheus itself (the dashboard) reports the error "context deadline exceeded" for the 2 node_exporter pods. When I create a single "curl" pod for curling ClusterIPs, the curl command hangs when trying to connect to these two.
So the question is how does one verify that flannel is functioning correctly?
helm install --name prometheus-service stable/prometheus
kubectl port-forward prometheus-service-server-<id> 9090
http://localhost:9090/targets (view in browser)
And see that some (perhaps all but one) of the node_exporter pods report "context deadline exceeded".