By aakarim
I can’t get Prometheus to pick up on my cAdvisor metrics on DOKS. It shows as a successful scrape on the Prometheus dashboard, but I can’t get any container_* metrics to show up in the query. Any ideas?
Here’s my job config for cadvisor:
- job_name: kubernetes-cadvisor
honor_timestamps: true
scrape_interval: 15s
scrape_timeout: 10s
metrics_path: /metrics
scheme: https
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
kubernetes_sd_configs:
- role: node
relabel_configs:
- separator: ;
regex: __meta_kubernetes_node_label_(.+)
replacement: $1
action: labelmap
- source_labels: [__address__]
separator: ;
regex: ([^:]+)(?::\d+)?
target_label: __address__
replacement: $1:10250
action: replace
- separator: ;
regex: (.*)
target_label: __metrics_path__
replacement: /metrics/cadvisor
action: replace
metric_relabel_configs:
- source_labels: [namespace]
separator: ;
regex: ^$
replacement: $1
action: drop
- source_labels: [pod_name]
separator: ;
regex: ^$
replacement: $1
action: drop
I’ve also tried the metric path as /metrics/cadvisor like some issues of Github have suggested.
This config was taken from the tutorial: https://www.digitalocean.com/community/tutorials/how-to-set-up-a-kubernetes-monitoring-stack-with-prometheus-grafana-and-alertmanager-on-digitalocean#step-6-—-configuring-the-monitoring-stack-optional
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.
Accepted Answer
pod_name and container_name were removed from cadvisor on 1.16. The metric_relabel_configs on pod_name is dropping the metrics. Try replace it with just pod.
See https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.16.md#removed-metrics
You can debug the target by going to the UI > Status > Targets. See if the target address is correct.
Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.
Full documentation for every DigitalOcean product.
The Wave has everything you need to know about building a business, from raising funding to marketing your product.
Scale up as you grow — whether you're running one virtual machine or ten thousand.

From GPU-powered inference and Kubernetes to managed databases and storage, get everything you need to build, scale, and deploy intelligent applications.
