I have three nodes in the cluster. Prometheus pods (which include server, alertmanager, node_exporter etc) start just fine. Unfortunately 2 of the 3 node_exporters cannot be reached. This seems like it must be some issue with flannel, but I don’t know how to begin to debug this.
Prometheus itself (the dashboard) reports the error “context deadline exceeded” for the 2 node_exporter pods. When I create a single “curl” pod for curling ClusterIPs, the curl command hangs when trying to connect to these two.
So the question is how does one verify that flannel is functioning correctly?
helm install --name prometheus-service stable/prometheus kubectl port-forward prometheus-service-server-<id> 9090 http://localhost:9090/targets (view in browser)
And see that some (perhaps all but one) of the node_exporter pods report “context deadline exceeded”.
These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.