Question

Failed to pull image from private repository on kubernetes cluster

Posted October 26, 2018 5.4k views
DigitalOceanDockerKubernetes

Hello,

I’m using DO’s new kubernetes cluster and it seems there’s an issue pulling an image form a private repository. I am using GitLab on the cluster with a configured docker registry. The registry works and I can pull the image locally. However when deploying on kubernetes I get the following error:

“Failed to pull image {redacted}: rpc error: code = Unknown desc = Error response from daemon: Get {redacted}: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)”

From related issues online it appears most people get this error with a proxy/dns issue so could this be an internal DO issue?

The request is not even logged on the registry server so it does not appear to be an authentication issue either. (Although I have a dockercfg secret setup for authentication)

Thanks,
Ryan

Add a comment
ryantwr
By:
ryantwr
Add a comment
5 comments
  • jayge November 2, 2018
    Reply Report

    I am also facing this issue. Is this something that you have figured out?

  • ryantwr November 8, 2018
    Reply Report

    Haven’t investigated much more, just avoiding it for now. However their support directed me to a previous FAQ of someone managing to setup gitlab + private repository deploying before. Not much help but I will probably go over my deployment again.

  • enlyt November 9, 2018
    Reply Report

    Did you configure the imagePullSecrets in the deployment yaml?
    https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/

  • richardio June 14, 2019
    Reply Report

    I’m having the exact same problem. I can pull images from any private docker registry outside of my cluster (eg dockerhub.io, gitlab.com etc). I can also push and pull images in private docker registries that I have created inside the kubernetes cluster (have created both a nexus registry and one using the ‘stable/docker-registry’ helm chart) from my local dev machine. The registries I have created in the cluster have a public https endpoint via a DigitalOcean load balancer.

    I also discovered that if I SSH into the actual nodes themselves and use docker commands directly on underlying droplet I get exactly the same result (external registries work fine, internal fail). Any help appreciated. I don’t want to admit how much sweat I have wasted on this one.

  • denisee35fd00e7058fc27d51a July 7, 2019
    Reply Report

    I am also having the exact same problem.One of my node can pull image and 2 of them can’t. I built a private repo inside k8 cluster.

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
4 answers
ai4 January 23, 2019

I had the same issue when running private registry on the DO cluster. I fixed it by scaling up registry to run on all nodes.
It seems to be a weird routing issue with the DO loadbalancer when addressing the outside IP from inside the cluster. DO k8s controller tries to be smart and routes node requests internally, instead of sending traffic to the loadbalancer IP. The issue though that if there is no registry service on that node, the packets go nowhere.
So, scale up or route through a non-DO LB.

Reply Report
  • PoslavskySV January 30, 2019

    Thanks for the answer. It’m facing exactly the same problem right now running k8s on Azure. It seems that they have exactly same issue. Your answer also explains why sometimes k8s successfully runs pod with image from in-cluster registry (when it starts on the same node with docker registry).

    Reply Report
PoslavskySV January 30, 2019
[deleted]
Reply Report
0x01 April 22, 2019

Any update on this ? Having the same issue here

Reply Report
stormofredrik September 12, 2019

Here’s what I did as a workaround:

  1. Set up your docker-registry ingress as usual (with tls etc.)
  2. Point your registry domain to your load balancer (as usual)
  3. Log in to your nodes via ssh
  4. Add the cluster IP of your ingress-controller to /etc/hosts
  <ingress controller IP> registry.example.com

Use the domain name when deploying images in the cluster (e.g registry.example.com/image)

If your /etc/hosts is managed you might need to update your /etc/cloud/templates/host.*.tmpl as well.

Reply Report
Submit an Answer

Related

Looking for something else?

Ask a new question Search for more help