Question

Kubernete Issue about CoreDNS i/o timeout.

coredns version : 1.9.4 kubernete version : 1.26.3-do.0

The coredns has no problem for a long time but It just cause this for a moment before everything go back to their normal. So I just try to figure out where is the cause.

I have two coredns pods on different node but Only one pod cause the error

2023-11-12T07:23:13.336588936Z [WARNING] No files matching import glob pattern: custom/*.override

2023-11-12T07:23:13.336638470Z [WARNING] No files matching import glob pattern: custom/*.server

2023-11-12T07:23:48.968921982Z [WARNING] No files matching import glob pattern: custom/*.override

2023-11-12T07:23:48.968955349Z [WARNING] No files matching import glob pattern: custom/*.server

2023-11-12T07:24:22.203655501Z [WARNING] No files matching import glob pattern: custom/*.override

2023-11-12T07:24:22.203724219Z [WARNING] No files matching import glob pattern: custom/*.server

2023-11-12T07:24:46.168260279Z [ERROR] plugin/errors: 2 externalDomain.com. A: read udp 10.244.1.107:44705->67.207.67.2:53: i/o timeout

2023-11-12T07:25:04.919158458Z [WARNING] No files matching import glob pattern: custom/*.override

2023-11-12T07:25:09.108144252Z [WARNING] No files matching import glob pattern: custom/*.server

2023-11-12T07:25:09.108152227Z [ERROR] plugin/errors: 2 externalDomain.biz. AAAA: read udp 10.244.1.107:54204->67.207.67.2:53: i/o timeout

2023-11-12T07:25:09.108156817Z [ERROR] plugin/errors: 2 externalDomain.biz. A: read udp 10.244.1.107:37713->67.207.67.2:53: i/o timeout

2023-11-12T07:25:09.108186396Z [ERROR] plugin/errors: 2 externalDomain.com. AAAA: read udp 10.244.1.107:48509->67.207.67.2:53: i/o timeout

2023-11-12T07:25:09.108191399Z [ERROR] plugin/errors: 2 externalDomain.com. AAAA: read udp 10.244.1.107:41671->67.207.67.2:53: i/o timeout

2023-11-12T07:25:09.108195499Z [ERROR] plugin/errors: 2 app-batchjob-service. A: read udp 10.244.1.107:55217->67.207.67.3:53: i/o timeout
2023-11-12T07:25:14.229263064Z [ERROR] plugin/errors: 2 externalDomain.com. A: read udp 10.244.1.107:57852->67.207.67.3:53: i/o timeout

2023-11-12T07:25:14.229357041Z [ERROR] plugin/errors: 2 logs.ap-southeast-1.amazonaws.com. AAAA: read udp 10.244.1.107:47403->67.207.67.3:53: i/o timeout

2023-11-12T07:25:14.229365269Z [ERROR] plugin/errors: 2 sqs.ap-southeast-1.amazonaws.com. A: read udp 10.244.1.107:34757->67.207.67.3:53: i/o timeout
2023-11-12T07:25:14.229369119Z [ERROR] plugin/errors: 2 sqs.ap-southeast-1.amazonaws.com. AAAA: read udp 10.244.1.107:54167->67.207.67.2:53: i/o timeout

2023-11-12T07:25:14.229373066Z [ERROR] plugin/errors: 2 api.telegram.org. AAAA: read udp 10.244.1.107:60622->67.207.67.3:53: i/o timeout
2023-11-12T07:25:14.229376620Z [ERROR] plugin/errors: 2 externalDomain.com. A: read udp 10.244.1.107:35431->67.207.67.3:53: i/o timeout

2023-11-12T07:25:14.229380502Z [ERROR] plugin/errors: 2 sqs.ap-southeast-1.amazonaws.com. A: read udp 10.244.1.107:33419->67.207.67.3:53: i/o timeout

2023-11-12T07:25:14.229383954Z [ERROR] plugin/errors: 2 app-batchjob-service. AAAA: read udp 10.244.1.107:60157->67.207.67.3:53: i/o timeout

2023-11-12T07:25:14.229888445Z [ERROR] plugin/errors: 2 logs.ap-southeast-1.amazonaws.com. AAAA: read udp 10.244.1.107:43677->67.207.67.2:53: i/o timeout

2023-11-12T07:25:14.229900169Z [ERROR] plugin/errors: 2 externalDomain.biz. AAAA: read udp 10.244.1.107:58469->67.207.67.2:53: i/o timeout

2023-11-12T07:25:14.229905418Z [ERROR] plugin/errors: 2 app-batchjob-important-service. A: read udp 10.244.1.107:58057->67.207.67.2:53: i/o timeout

2023-11-12T07:25:14.229910647Z [ERROR] plugin/errors: 2 sqs.ap-southeast-1.amazonaws.com. AAAA: read udp 10.244.1.107:34731->67.207.67.2:53: i/o timeout

2023-11-12T07:25:14.229915901Z [ERROR] plugin/errors: 2 app-batchjob-service. AAAA: read udp 10.244.1.107:43181->67.207.67.2:53: i/o timeout

2023-11-12T07:25:14.229921161Z [ERROR] plugin/errors: 2 app-batchjob-service. A: read udp 10.244.1.107:34212->67.207.67.3:53: i/o timeout

2023-11-12T07:25:14.229926497Z [ERROR] plugin/errors: 2 app2.externalDomain.com. A: read udp 10.244.1.107:50008->67.207.67.2:53: i/o timeout

2023-11-12T07:25:14.229931779Z [ERROR] plugin/errors: 2 externalDomain.com. A: read udp 10.244.1.107:51822->67.207.67.2:53: i/o timeout

2023-11-12T07:25:17.570079729Z [WARNING] plugin/health: Local health request to "http://:8080/health" failed: Get "http://:8080/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

2023-11-12T07:25:20.266071569Z [ERROR] plugin/errors: 2 app2.externalDomain.net. AAAA: read udp 10.244.1.107:48985->67.207.67.2:53: i/o timeout

2023-11-12T07:25:20.571310265Z [WARNING] plugin/health: Local health request to "http://:8080/health" failed: Get "http://:8080/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

2023-11-12T07:25:23.492183926Z [WARNING] plugin/health: Local health request to "http://:8080/health" took more than 1s: 2.920673076s

2023-11-12T07:25:29.483945199Z [ERROR] plugin/errors: 2 kubernetes.github.io. AAAA: read udp 10.244.1.107:60190->67.207.67.3:53: i/o timeout

2023-11-12T07:25:31.068608496Z [WARNING] No files matching import glob pattern: custom/*.override

2023-11-12T07:25:57.346783290Z [WARNING] No files matching import glob pattern: custom/*.server

2023-11-12T07:26:00.644749460Z [ERROR] plugin/errors: 2 api.externalDomain.com. A: read udp 10.244.1.107:42706->67.207.67.3:53: i/o timeout

2023-11-12T07:26:09.475053715Z [ERROR] plugin/errors: 2 logs.ap-southeast-1.amazonaws.com. A: read udp 10.244.1.107:34931->67.207.67.3:53: i/o timeout

2023-11-12T07:26:09.475212221Z [ERROR] plugin/errors: 2 app-batchjob-service. AAAA: read udp 10.244.1.107:53323->67.207.67.3:53: i/o timeout

2023-11-12T07:26:09.475222140Z [WARNING] No files matching import glob pattern: custom/*.override

2023-11-12T07:26:09.475227796Z [WARNING] No files matching import glob pattern: custom/*.server

2023-11-12T07:26:09.475232601Z [ERROR] plugin/errors: 2 db.ondigitalocean.com. A: read udp 10.244.1.107:40380->67.207.67.3:53: i/o timeout

2023-11-12T07:26:12.308624102Z [INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: watch of *v1.Service ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding

2023-11-12T07:26:12.308672715Z [INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: watch of *v1.EndpointSlice ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding

2023-11-12T07:26:12.308678891Z [INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: watch of *v1.Namespace ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding

2023-11-12T07:26:21.066646378Z [ERROR] plugin/errors: 2 app-batchjob-commission-service. A: read udp 10.244.1.107:45214->67.207.67.2:53: i/o timeout

2023-11-12T07:26:31.090310738Z [ERROR] plugin/errors: 2 app-batchjob-service. AAAA: read udp 10.244.1.107:57194->67.207.67.3:53: i/o timeout
2023-11-12T07:26:39.935621019Z [WARNING] No files matching import glob pattern: custom/*.override

2023-11-12T07:26:39.935672547Z [WARNING] No files matching import glob pattern: custom/*.server

2023-11-12T07:27:32.840471455Z [ERROR] plugin/errors: 2 xx.cloudfront.net. AAAA: read udp 10.244.1.107:50651->67.207.67.2:53: i/o timeout

2023-11-12T07:27:32.841015219Z [ERROR] plugin/errors: 2 sqs.ap-southeast-1.amazonaws.com. AAAA: read udp 10.244.1.107:53995->67.207.67.3:53: i/o timeout

2023-11-12T07:27:32.841023509Z [ERROR] plugin/errors: 2 logs.ap-southeast-1.amazonaws.com. AAAA: read udp 10.244.1.107:59555->67.207.67.2:53: i/o timeout

2023-11-12T07:27:32.841029028Z [ERROR] plugin/errors: 2 externalDomain.com. A: read udp 10.244.1.107:50236->67.207.67.2:53: i/o timeout
2023-11-12T07:27:32.841036830Z [ERROR] plugin/errors: 2 db.ondigitalocean.com. AAAA: read udp 10.244.1.107:59717->67.207.67.2:53: i/o timeout

2023-11-12T07:27:32.841059424Z [ERROR] plugin/errors: 2 externalDomain.com. AAAA: read udp 10.244.1.107:34569->67.207.67.3:53: i/o timeout
2023-11-12T07:27:32.841066340Z [WARNING] No files matching import glob pattern: custom/*.override

2023-11-12T07:27:32.841073041Z [WARNING] No files matching import glob pattern: custom/*.server

Submit an answer


This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

Sign In or Sign Up to Answer

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

Bobby Iliev
Site Moderator
Site Moderator badge
November 23, 2023

Hey @thanakornscubadiver,

I’ve noticed that there was an incident reported on the status page affecting some DOKS clusters:

https://status.digitalocean.com/incidents/wpz7gdly2p32

This has now been resolved, but if you are still seeing the problem, please reach out to the DigitalOcean support team who might be able to investigate this further for you:

https://www.digitalocean.com/support/

Hope that helps!

- Bobby.

Try DigitalOcean for free

Click below to sign up and get $200 of credit to try our products over 60 days!

Sign up

card icon
Get our biweekly newsletter

Sign up for Infrastructure as a Newsletter.

Sign up
card icon
Hollie's Hub for Good

Working on improving health and education, reducing inequality, and spurring economic growth? We’d like to help.

Learn more
card icon
Become a contributor

You get paid; we donate to tech nonprofits.

Learn more
Welcome to the developer cloud

DigitalOcean makes it simple to launch in the cloud and scale up as you grow – whether you’re running one virtual machine or ten thousand.

Learn more ->
DigitalOcean Cloud Control Panel