Question
I can no longer connect to external URLs after cluster upgrade
- Posted January 14, 2020
- Kubernetes
I upgraded my k8 cluster a few days ago and since then I have not been able to connect to external URLs.
I am getting a ‘connect timed out’ error when trying to load an external site and locally this same code is running properly.
Do I need to do something special to allow my pod to communicate externally? I messed around with some network policy stuff but that didn’t seem to help.
Could it be related to this change/fix? 1.16.2-do.2 (2020-01-10) :: Kubernetes Changelog
These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.
Ok to add more info this is only happening when trying to connect to sites that are running on the same cluster.
I run a test pod like this
kubectl run -it --rm --restart=Never alpine --image=alpine sh
And wget google.com works while wget brownbear.tech hangs.
Locally on my machine wget brownbear.tech works as expected along with following the HSTS policy to redirect to https.
It seems like the issue is a networking issue where it cannot properly route a URL which is also hosted on the cluster. For one of my sites hosted on a different cluster I do not see the issue making the connection.
kube-system all look healthy. This is happening in a cronjob and it has failed on each run/pod consistently since the cluster upgrade on Monday evening. Resources utilization on the cluster also looks good.
Do you see all the pods in your kubesystem running healthy? I would namely look for coredns, kube-proxy, and cilium. If not I would ensure they have enough resources to run reliably and try kicking the pods again. Is the connection issue happening from all nodes?