By aakarim
I can’t get Prometheus to pick up on my cAdvisor metrics on DOKS. It shows as a successful scrape on the Prometheus dashboard, but I can’t get any container_* metrics to show up in the query. Any ideas?
Here’s my job config for cadvisor:
- job_name: kubernetes-cadvisor
honor_timestamps: true
scrape_interval: 15s
scrape_timeout: 10s
metrics_path: /metrics
scheme: https
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
kubernetes_sd_configs:
- role: node
relabel_configs:
- separator: ;
regex: __meta_kubernetes_node_label_(.+)
replacement: $1
action: labelmap
- source_labels: [__address__]
separator: ;
regex: ([^:]+)(?::\d+)?
target_label: __address__
replacement: $1:10250
action: replace
- separator: ;
regex: (.*)
target_label: __metrics_path__
replacement: /metrics/cadvisor
action: replace
metric_relabel_configs:
- source_labels: [namespace]
separator: ;
regex: ^$
replacement: $1
action: drop
- source_labels: [pod_name]
separator: ;
regex: ^$
replacement: $1
action: drop
I’ve also tried the metric path as /metrics/cadvisor like some issues of Github have suggested.
This config was taken from the tutorial: https://www.digitalocean.com/community/tutorials/how-to-set-up-a-kubernetes-monitoring-stack-with-prometheus-grafana-and-alertmanager-on-digitalocean#step-6-—-configuring-the-monitoring-stack-optional
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
Accepted Answer
pod_name
and container_name
were removed from cadvisor on 1.16. The metric_relabel_configs
on pod_name
is dropping the metrics. Try replace it with just pod
.
See https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.16.md#removed-metrics
You can debug the target by going to the UI > Status > Targets. See if the target address is correct.
Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.
Full documentation for every DigitalOcean product.
The Wave has everything you need to know about building a business, from raising funding to marketing your product.
Stay up to date by signing up for DigitalOcean’s Infrastructure as a Newsletter.
New accounts only. By submitting your email you agree to our Privacy Policy
Scale up as you grow — whether you're running one virtual machine or ten thousand.
Sign up and get $200 in credit for your first 60 days with DigitalOcean.*
*This promotional offer applies to new accounts only.