Question

kubedns endpoint is not working and kube-dns cannot resolve any services.

Posted December 21, 2019 6.5k views
KubernetesDigitalOcean Managed Kubernetes

I’m using Digital Ocean kubernetes.
It had been working well but now it cannot resolve any services.

It seems that the kube-dns endpoint resource is someting wrong.

Here is my logs.

$ for p in $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name); do kubectl logs --namespace=kube-system $p; done
.:53
2019-11-30T10:15:04.248Z [INFO] plugin/reload: Running configuration MD5 = 68ad62e82bb7a01ce4a04dd06bd0bf2c
2019-11-30T10:15:04.248Z [INFO] CoreDNS-1.6.2
2019-11-30T10:15:04.248Z [INFO] linux/amd64, go1.12.8, 795a3eb
CoreDNS-1.6.2
linux/amd64, go1.12.8, 795a3eb
2019-12-15T06:25:22.661Z [ERROR] plugin/errors: 2 sgp1.sonar.digitalocean.com. AAAA: read udp 10.244.0.114:50603->67.207.67.2:53: i/o timeout
.:53
2019-11-30T10:15:05.293Z [INFO] plugin/reload: Running configuration MD5 = 68ad62e82bb7a01ce4a04dd06bd0bf2c
2019-11-30T10:15:05.294Z [INFO] CoreDNS-1.6.2
2019-11-30T10:15:05.294Z [INFO] linux/amd64, go1.12.8, 795a3eb
CoreDNS-1.6.2
linux/amd64, go1.12.8, 795a3eb
2019-12-15T06:25:11.969Z [ERROR] plugin/errors: 2 8690ff65-fea8-4c25-9376-7b2d633d2245.internal.k8s.ondigitalocean.com. A: read udp 10.244.0.148:48223->67.207.67.3:53: i/o timeout
2019-12-15T06:25:11.928Z [ERROR] plugin/errors: 2 8690ff65-fea8-4c25-9376-7b2d633d2245.internal.k8s.ondigitalocean.com. AAAA: read udp 10.244.0.148:40733->67.207.67.3:53: i/o timeout
2019-12-15T06:25:17.475Z [ERROR] plugin/errors: 2 sgp1.sonar.digitalocean.com. A: read udp 10.244.0.148:44810->67.207.67.2:53: i/o timeout
2019-12-15T06:25:17.475Z [ERROR] plugin/errors: 2 sgp1.sonar.digitalocean.com. AAAA: read udp 10.244.0.148:33077->67.207.67.2:53: i/o timeout

and kube-dns service is following.

$ kubectl get svc --namespace=kube-system
NAME            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE
kube-dns        ClusterIP   10.245.0.10     <none>        53/UDP,53/TCP,9153/TCP   20d
$ kubectl get ep kube-dns --namespace=kube-system
NAME       ENDPOINTS   AGE
kube-dns               20d
$ kubectl describe ep kube-dns --namespace=kube-system
Name:         kube-dns
Namespace:    kube-system
Labels:       k8s-app=kube-dns
              kubernetes.io/cluster-service=true
              kubernetes.io/name=CoreDNS
Annotations:  <none>
Subsets:
  Addresses:          <none>
  NotReadyAddresses:  10.244.0.114,10.244.0.148
  Ports:
    Name     Port  Protocol
    ----     ----  --------
    dns      53    UDP
    dns-tcp  53    TCP
    metrics  9153  TCP

Events:  <none>
$ kubectl get po coredns-84c79f5fb4-k8crt --namespace=kube-system
NAME                       READY   STATUS    RESTARTS   AGE
coredns-84c79f5fb4-k8crt   1/1     Running   0          20d
$ kubectl describe po coredns-84c79f5fb4-k8crt --namespace=kube-system
Name:                 coredns-84c79f5fb4-k8crt
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 pool-d2o-6v7u/10.130.13.143
Start Time:           Sat, 30 Nov 2019 19:15:01 +0900
Labels:               k8s-app=kube-dns
                      pod-template-hash=84c79f5fb4
Annotations:          <none>
Status:               Running
IP:                   10.244.0.114
IPs:
  IP:           10.244.0.114
Controlled By:  ReplicaSet/coredns-84c79f5fb4
Containers:
  coredns:
    Container ID:  docker://173d34004f06db3df388f0e6c8b2d7f72ae4c4adff286e664c3a8b5bb24f8f8e
    Image:         docker.io/coredns/coredns:1.6.2
    Image ID:      docker-pullable://coredns/coredns@sha256:12eb885b8685b1b13a04ecf5c23bc809c2e57917252fd7b0be9e9c00644e8ee5
    Ports:         53/UDP, 53/TCP, 9153/TCP
    Host Ports:    0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    State:          Running
      Started:      Sat, 30 Nov 2019 19:15:03 +0900
    Ready:          True
    Restart Count:  0
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
    Readiness:    http-get http://:8181/ready delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from coredns-token-82b8t (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
  coredns-token-82b8t:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  coredns-token-82b8t
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  beta.kubernetes.io/os=linux
Tolerations:     CriticalAddonsOnly
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

edited by bobbyiliev

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
Submit an Answer
1 answer