do-agent - Client.Timeout exceeded while awaiting headers

January 13, 2018 874 views
Monitoring Ubuntu 16.04

Hi,

Our droplets in AMS3 are getting constant timeouts when trying to send metrics. This is really filling up our logs.

Is anybody else experiencing this?

All outbound traffic is being allowed from the droplets.

Jan 13 12:25:36 hostx syslog do-agent[1451]: 2018/01/13 11:25:35 Sending metrics to DigitalOcean: Post https://ams3.sonar.digitalocean.com/v1/metrics/droplet_id/XXXXXXX: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Jan 13 12:26:50 hostx syslog do-agent[1451]: 2018/01/13 11:26:49 Sending metrics to DigitalOcean: Post https://ams3.sonar.digitalocean.com/v1/metrics/droplet_id/XXXXXXX: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

3 Answers

i saw something similar when we had problems with DNS or IPv6 settings.
try dig -4 ams3.sonar.digitalocean.com for DNS query,
and ping -c 5 ams3.sonar.digitalocean.com for ping with DNS query,
and ping -c 5 162.243.188.200 for ping without DNS query.

IP 162.243.188.200 stands for ams3.sonar.digitalocean.com .

Certainly no problems here, except for a pretty big delay in pings:


; <<>> DiG 9.10.3-P4-Ubuntu <<>> -4 ams3.sonar.digitalocean.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43050
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;ams3.sonar.digitalocean.com.   IN  A

;; ANSWER SECTION:
ams3.sonar.digitalocean.com. 126 IN A   162.243.188.200

;; Query time: 0 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Sat Jan 13 14:49:59 UTC 2018
;; MSG SIZE  rcvd: 72

PING ams3.sonar.digitalocean.com (162.243.188.200) 56(84) bytes of data.
64 bytes from 162.243.188.200: icmp_seq=1 ttl=59 time=80.4 ms
64 bytes from 162.243.188.200: icmp_seq=2 ttl=59 time=85.1 ms
64 bytes from 162.243.188.200: icmp_seq=3 ttl=59 time=80.6 ms
64 bytes from 162.243.188.200: icmp_seq=4 ttl=59 time=79.4 ms
64 bytes from 162.243.188.200: icmp_seq=5 ttl=59 time=79.4 ms

PING 162.243.188.200 (162.243.188.200) 56(84) bytes of data.
64 bytes from 162.243.188.200: icmp_seq=1 ttl=59 time=86.8 ms
64 bytes from 162.243.188.200: icmp_seq=2 ttl=59 time=84.5 ms
64 bytes from 162.243.188.200: icmp_seq=3 ttl=59 time=79.4 ms
64 bytes from 162.243.188.200: icmp_seq=4 ttl=59 time=79.4 ms
64 bytes from 162.243.188.200: icmp_seq=5 ttl=59 time=91.6 ms

Looks like this was related to the incident here - https://status.digitalocean.com/incidents/y7dl42h08t4x .

If so, you should not be seeing the issue anymore.

Have another answer? Share your knowledge.