Question
Kubernete Issue about CoreDNS i/o timeout.
- Posted on November 12, 2023
- KubernetesNetworking
Asked by Thanakorn
coredns version : 1.9.4 kubernete version : 1.26.3-do.0
The coredns has no problem for a long time but It just cause this for a moment before everything go back to their normal. So I just try to figure out where is the cause.
I have two coredns pods on different node but Only one pod cause the error
2023-11-12T07:23:13.336588936Z [WARNING] No files matching import glob pattern: custom/*.override
2023-11-12T07:23:13.336638470Z [WARNING] No files matching import glob pattern: custom/*.server
2023-11-12T07:23:48.968921982Z [WARNING] No files matching import glob pattern: custom/*.override
2023-11-12T07:23:48.968955349Z [WARNING] No files matching import glob pattern: custom/*.server
2023-11-12T07:24:22.203655501Z [WARNING] No files matching import glob pattern: custom/*.override
2023-11-12T07:24:22.203724219Z [WARNING] No files matching import glob pattern: custom/*.server
2023-11-12T07:24:46.168260279Z [ERROR] plugin/errors: 2 externalDomain.com. A: read udp 10.244.1.107:44705->67.207.67.2:53: i/o timeout
2023-11-12T07:25:04.919158458Z [WARNING] No files matching import glob pattern: custom/*.override
2023-11-12T07:25:09.108144252Z [WARNING] No files matching import glob pattern: custom/*.server
2023-11-12T07:25:09.108152227Z [ERROR] plugin/errors: 2 externalDomain.biz. AAAA: read udp 10.244.1.107:54204->67.207.67.2:53: i/o timeout
2023-11-12T07:25:09.108156817Z [ERROR] plugin/errors: 2 externalDomain.biz. A: read udp 10.244.1.107:37713->67.207.67.2:53: i/o timeout
2023-11-12T07:25:09.108186396Z [ERROR] plugin/errors: 2 externalDomain.com. AAAA: read udp 10.244.1.107:48509->67.207.67.2:53: i/o timeout
2023-11-12T07:25:09.108191399Z [ERROR] plugin/errors: 2 externalDomain.com. AAAA: read udp 10.244.1.107:41671->67.207.67.2:53: i/o timeout
2023-11-12T07:25:09.108195499Z [ERROR] plugin/errors: 2 app-batchjob-service. A: read udp 10.244.1.107:55217->67.207.67.3:53: i/o timeout
2023-11-12T07:25:14.229263064Z [ERROR] plugin/errors: 2 externalDomain.com. A: read udp 10.244.1.107:57852->67.207.67.3:53: i/o timeout
2023-11-12T07:25:14.229357041Z [ERROR] plugin/errors: 2 logs.ap-southeast-1.amazonaws.com. AAAA: read udp 10.244.1.107:47403->67.207.67.3:53: i/o timeout
2023-11-12T07:25:14.229365269Z [ERROR] plugin/errors: 2 sqs.ap-southeast-1.amazonaws.com. A: read udp 10.244.1.107:34757->67.207.67.3:53: i/o timeout
2023-11-12T07:25:14.229369119Z [ERROR] plugin/errors: 2 sqs.ap-southeast-1.amazonaws.com. AAAA: read udp 10.244.1.107:54167->67.207.67.2:53: i/o timeout
2023-11-12T07:25:14.229373066Z [ERROR] plugin/errors: 2 api.telegram.org. AAAA: read udp 10.244.1.107:60622->67.207.67.3:53: i/o timeout
2023-11-12T07:25:14.229376620Z [ERROR] plugin/errors: 2 externalDomain.com. A: read udp 10.244.1.107:35431->67.207.67.3:53: i/o timeout
2023-11-12T07:25:14.229380502Z [ERROR] plugin/errors: 2 sqs.ap-southeast-1.amazonaws.com. A: read udp 10.244.1.107:33419->67.207.67.3:53: i/o timeout
2023-11-12T07:25:14.229383954Z [ERROR] plugin/errors: 2 app-batchjob-service. AAAA: read udp 10.244.1.107:60157->67.207.67.3:53: i/o timeout
2023-11-12T07:25:14.229888445Z [ERROR] plugin/errors: 2 logs.ap-southeast-1.amazonaws.com. AAAA: read udp 10.244.1.107:43677->67.207.67.2:53: i/o timeout
2023-11-12T07:25:14.229900169Z [ERROR] plugin/errors: 2 externalDomain.biz. AAAA: read udp 10.244.1.107:58469->67.207.67.2:53: i/o timeout
2023-11-12T07:25:14.229905418Z [ERROR] plugin/errors: 2 app-batchjob-important-service. A: read udp 10.244.1.107:58057->67.207.67.2:53: i/o timeout
2023-11-12T07:25:14.229910647Z [ERROR] plugin/errors: 2 sqs.ap-southeast-1.amazonaws.com. AAAA: read udp 10.244.1.107:34731->67.207.67.2:53: i/o timeout
2023-11-12T07:25:14.229915901Z [ERROR] plugin/errors: 2 app-batchjob-service. AAAA: read udp 10.244.1.107:43181->67.207.67.2:53: i/o timeout
2023-11-12T07:25:14.229921161Z [ERROR] plugin/errors: 2 app-batchjob-service. A: read udp 10.244.1.107:34212->67.207.67.3:53: i/o timeout
2023-11-12T07:25:14.229926497Z [ERROR] plugin/errors: 2 app2.externalDomain.com. A: read udp 10.244.1.107:50008->67.207.67.2:53: i/o timeout
2023-11-12T07:25:14.229931779Z [ERROR] plugin/errors: 2 externalDomain.com. A: read udp 10.244.1.107:51822->67.207.67.2:53: i/o timeout
2023-11-12T07:25:17.570079729Z [WARNING] plugin/health: Local health request to "http://:8080/health" failed: Get "http://:8080/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2023-11-12T07:25:20.266071569Z [ERROR] plugin/errors: 2 app2.externalDomain.net. AAAA: read udp 10.244.1.107:48985->67.207.67.2:53: i/o timeout
2023-11-12T07:25:20.571310265Z [WARNING] plugin/health: Local health request to "http://:8080/health" failed: Get "http://:8080/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2023-11-12T07:25:23.492183926Z [WARNING] plugin/health: Local health request to "http://:8080/health" took more than 1s: 2.920673076s
2023-11-12T07:25:29.483945199Z [ERROR] plugin/errors: 2 kubernetes.github.io. AAAA: read udp 10.244.1.107:60190->67.207.67.3:53: i/o timeout
2023-11-12T07:25:31.068608496Z [WARNING] No files matching import glob pattern: custom/*.override
2023-11-12T07:25:57.346783290Z [WARNING] No files matching import glob pattern: custom/*.server
2023-11-12T07:26:00.644749460Z [ERROR] plugin/errors: 2 api.externalDomain.com. A: read udp 10.244.1.107:42706->67.207.67.3:53: i/o timeout
2023-11-12T07:26:09.475053715Z [ERROR] plugin/errors: 2 logs.ap-southeast-1.amazonaws.com. A: read udp 10.244.1.107:34931->67.207.67.3:53: i/o timeout
2023-11-12T07:26:09.475212221Z [ERROR] plugin/errors: 2 app-batchjob-service. AAAA: read udp 10.244.1.107:53323->67.207.67.3:53: i/o timeout
2023-11-12T07:26:09.475222140Z [WARNING] No files matching import glob pattern: custom/*.override
2023-11-12T07:26:09.475227796Z [WARNING] No files matching import glob pattern: custom/*.server
2023-11-12T07:26:09.475232601Z [ERROR] plugin/errors: 2 db.ondigitalocean.com. A: read udp 10.244.1.107:40380->67.207.67.3:53: i/o timeout
2023-11-12T07:26:12.308624102Z [INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: watch of *v1.Service ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
2023-11-12T07:26:12.308672715Z [INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: watch of *v1.EndpointSlice ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
2023-11-12T07:26:12.308678891Z [INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: watch of *v1.Namespace ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
2023-11-12T07:26:21.066646378Z [ERROR] plugin/errors: 2 app-batchjob-commission-service. A: read udp 10.244.1.107:45214->67.207.67.2:53: i/o timeout
2023-11-12T07:26:31.090310738Z [ERROR] plugin/errors: 2 app-batchjob-service. AAAA: read udp 10.244.1.107:57194->67.207.67.3:53: i/o timeout
2023-11-12T07:26:39.935621019Z [WARNING] No files matching import glob pattern: custom/*.override
2023-11-12T07:26:39.935672547Z [WARNING] No files matching import glob pattern: custom/*.server
2023-11-12T07:27:32.840471455Z [ERROR] plugin/errors: 2 xx.cloudfront.net. AAAA: read udp 10.244.1.107:50651->67.207.67.2:53: i/o timeout
2023-11-12T07:27:32.841015219Z [ERROR] plugin/errors: 2 sqs.ap-southeast-1.amazonaws.com. AAAA: read udp 10.244.1.107:53995->67.207.67.3:53: i/o timeout
2023-11-12T07:27:32.841023509Z [ERROR] plugin/errors: 2 logs.ap-southeast-1.amazonaws.com. AAAA: read udp 10.244.1.107:59555->67.207.67.2:53: i/o timeout
2023-11-12T07:27:32.841029028Z [ERROR] plugin/errors: 2 externalDomain.com. A: read udp 10.244.1.107:50236->67.207.67.2:53: i/o timeout
2023-11-12T07:27:32.841036830Z [ERROR] plugin/errors: 2 db.ondigitalocean.com. AAAA: read udp 10.244.1.107:59717->67.207.67.2:53: i/o timeout
2023-11-12T07:27:32.841059424Z [ERROR] plugin/errors: 2 externalDomain.com. AAAA: read udp 10.244.1.107:34569->67.207.67.3:53: i/o timeout
2023-11-12T07:27:32.841066340Z [WARNING] No files matching import glob pattern: custom/*.override
2023-11-12T07:27:32.841073041Z [WARNING] No files matching import glob pattern: custom/*.server
Submit an answer
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.
Featured on Community
Get our biweekly newsletter
Sign up for Infrastructure as a Newsletter.
Hollie's Hub for Good
Working on improving health and education, reducing inequality, and spurring economic growth? We'd like to help.
Become a contributor
Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.
Hey @thanakornscubadiver,
I’ve noticed that there was an incident reported on the status page affecting some DOKS clusters:
This has now been resolved, but if you are still seeing the problem, please reach out to the DigitalOcean support team who might be able to investigate this further for you:
Hope that helps!
- Bobby.