cAdvisor on Prometheus with Kubernetes

Question

I can’t get Prometheus to pick up on my cAdvisor metrics on DOKS. It shows as a successful scrape on the Prometheus dashboard, but I can’t get any container_* metrics to show up in the query. Any ideas?

Here’s my job config for cadvisor:

    - job_name: kubernetes-cadvisor
      honor_timestamps: true
      scrape_interval: 15s
      scrape_timeout: 10s
      metrics_path: /metrics
      scheme: https
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - separator: ;
        regex: __meta_kubernetes_node_label_(.+)
        replacement: $1
        action: labelmap
      - source_labels: [__address__]
        separator: ;
        regex: ([^:]+)(?::\d+)?
        target_label: __address__
        replacement: $1:10250
        action: replace
      - separator: ;
        regex: (.*)
        target_label: __metrics_path__
        replacement: /metrics/cadvisor
        action: replace
      metric_relabel_configs:
      - source_labels: [namespace]
        separator: ;
        regex: ^$
        replacement: $1
        action: drop
      - source_labels: [pod_name]
        separator: ;
        regex: ^$
        replacement: $1
        action: drop

I’ve also tried the metric path as /metrics/cadvisor like some issues of Github have suggested.

This config was taken from the tutorial: https://www.digitalocean.com/community/tutorials/how-to-set-up-a-kubernetes-monitoring-stack-with-prometheus-grafana-and-alertmanager-on-digitalocean#step-6-—-configuring-the-monitoring-stack-optional

nmpacheco · Accepted Answer

pod_name and container_name were removed from cadvisor on 1.16. The metric_relabel_configs on pod_name is dropping the metrics. Try replace it with just pod. See https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.16.md#removed-metrics

dhanabalankct · Answer

try using /metrics.It might work.

vodangkhoa873 · Answer

You can debug the target by going to the UI > Status > Targets.See if the target address is correct.

aakarim · Answer

It does work if I try the query to the API server proxy:kubectl get --raw “/api/v1/nodes/kafka-9utff1mmv-83tab/proxy/metrics/cadvisor” # HELP cadvisor_version_info A metric with a constant '1' value labeled by kernel version, OS version, docker version, cadvisor version & cadvisor revision. # TYPE cadvisor_version_info gauge cadvisor_version_info{cadvisorRevision="",cadvisorVersion="",dockerVersion="19.03.13",kernelVersion="4.19.0-11-amd64",osVersion="Debian GNU/Linux 10 (buster)"} 1 # HELP container_cpu_cfs_periods_total Number of elapsed enforcement period intervals. # TYPE container_cpu_cfs_periods_total counter container_cpu_cfs_periods_total{container="",id="/kubepods/burstable/pod6347fe45-177c-4a6f-b0e9-a69ab700b25a",image="",name="",namespace="kube-system",pod="do-node-agent-qk929"} 29764 1613511629268 container_cpu_cfs_periods_total{container="",id="/kubepods/burstable/podd5e10ede-bcfe-4d91-aa0a-eaa19605178a",image="",name="",namespace="kafka",pod="strimzi-cluster-operator-68c6747bc6-gqwhs"} 1.114783e+06 1613511638725 container_cpu_cfs_periods_total{container="",id="/kubepods/pod5c1288c3-4f95-43ca-a2b8-4cf13b727736",image="",name="",namespace="monitoring",pod="lix-cluster-observability-node-exporter-md459"} 2.0123e+06 1613511631191 container_cpu_cfs_periods_total{container="addon-resizer",id="/kubepods/burstable/pod44a92bf5-08c9-4cb3-94b5-2cdfc330b570/c90c453056663e1b5c7b38504b77533f2bad2ba72f293c3caec52c5425055ee6",image="sha256:9b0815c8711889802a3081d1a609cc4251357e6ec0a28ac5963aac72bec67691",name="k8s_addon-resizer_lix-cluster-observability-kube-state-metrics-777d57f84b-46z4n_monitoring_44a92bf5-08c9-4cb3-94b5-2cdfc330b570_0",namespace="monitoring",pod="lix-cluster-observability-kube-state-metrics-777d57f84b-46z4n"} 1111 1613511638651 container_cpu_cfs_periods_total{container="do-node-agent",id="/kubepods/burstable/pod6347fe45-177c-4a6f-b0e9-a69ab700b25a/edf80465a6f677870115563d2544b8e5cee5033e9d7fa5f16c924e99ba8ed1e0",image="sha256:bd21a39d87de1a674a19ca48c05e34623f09e51ba75246f857ede8dfda62f3d8",name="k8s_do-node-agent_do-node-agent-qk929_kube-system_6347fe45-177c-4a6f-b0e9-a69ab700b25a_0",namespace="kube-system",pod="do-node-agent-qk929"} 29689 1613511633511 container_cpu_cfs_periods_total{container="prometheus-node-exporter",id="/kubepods/pod5c1288c3-4f95-43ca-a2b8-4cf13b727736/42d82062d336b2bad8a1d2b5ed66```

Related

Question

cAdvisor on Prometheus with Kubernetes

Related

Try DigitalOcean for free