Related

Join the DigitalOcean Community Community

Join 1M+ other developers and:

  • Get help and share knowledge in Q&A
  • Subscribe to topics of interest
  • Get courses & tools that help you grow as a developer or small business owner
 Join Now
The Kubernetes Service and Docker Private Registries Question Question
On one of the (newly created) Kubernetes nodes, all pods with private docker image fail with ImagePullBackOff Question Question

Question

DO k8s pull from private registry

Posted May 17, 2020 2.3k views
DockerKubernetes

Hi, has anyone come across this type of issue:
I am trying to setup k8s cluster and I also have a private registry running on a droplet, when I try to deploy an image from my private registry I get this error

Error response from daemon: Get https://artifacotry_ip: http: server gave HTTP response to HTTPS client

ok so I thought I am gonna setup a reverse proxy and add a self-signed certificate to it, you will think yeah this should solve the issue above, which it did, BUT it brings a new issues instead when I deploy again

Error response from daemon: Get https://artifactory_ip: x509: certificate signed by unknown authority

so because k8s is a managed service of digitalocean, I don’t have access to master node to push my certificates there and as you know I cannot ssh to k8s droplets either.

Anyone has any idea how to solve this issue?

Thank you.

Add a comment
MBII
By:
MBII

Related

Join the DigitalOcean Community Community

Join 1M+ other developers and:

  • Get help and share knowledge in Q&A
  • Subscribe to topics of interest
  • Get courses & tools that help you grow as a developer or small business owner
 Join Now
The Kubernetes Service and Docker Private Registries Question Question
On one of the (newly created) Kubernetes nodes, all pods with private docker image fail with ImagePullBackOff Question Question
Add a comment
1 comment
  • argadeis May 18, 2020
    Reply Report

    @jkwiatkoski
    is it not possible for the K8s cluster to pull images from a private container registry running in a droplet in the same VPC ?
    Assuming we have setup TLS on the Private Container Registry using a self-signed cert?
    So we would have to tell the K8s cluster to use that self-signed cert to do the image pull…

    Thanks in advance

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
Submit an Answer
4 answers
nabsul May 18, 2020

This doesn’t answer your question directly, but have you considered using a Docker registry service to host your images? I’ve found using AWS ECR to be much less of a headache than hosting the images myself.

If you do want to consider that route, I wrote a tool the connect a Digital Ocean K8s cluster to AWS ECR: https://github.com/nabsul/k8s-ecr-login-renew

Reply Report
jkwiatkoski May 18, 2020

Alternatively you could leverage our DO Container registry! Docs here: https://www.digitalocean.com/docs/images/container-registry/quickstart/

Reply Report
jkwiatkoski May 18, 2020

Hi there,

Our DOKS nodes are configured to reject pulling from any non secured registries. A simple solution would be to secure your registry with an SSL cert perhaps from https://letsencrypt.org/

Regards,

John Kwiatkoski
Senior Developer Support Engineer - Kubernetes

Reply Report
  • argadeis May 18, 2020

    @davidpesticcio @jkwiatkoski
    Thanks for your answers.

    So is it not possible for the K8s cluster to pull images from a private container registry running in a droplet in the same VPC ?
    Assuming we have setup TLS on the Private Container Registry using a self-signed cert?
    So we would have to tell the K8s cluster to use that self-signed cert to do the image pull…

    Thanks in advance

    Reply Report
    • jkwiatkoski May 18, 2020

      You certainly can pull an image from a different VPC.Image pulls are not restricted to a VPC.

      The issue here is a self signed certificate is by definition insecure. The registry would have to be properly secured (SSL) if you want your DOKS nodes to be able to pull it.

      Reply Report
      • argadeis May 18, 2020

        sorry to bug again, I just want to understand thou, for lets encrypt to sign a cert for me, my domain needs to be public yes?
        I am using a droplet though that has a private IP and not a registered Public DNS,

        So I cannot see how I could obtain a TLS connection between DOKS and the droplet that runs the Registry, with a LetsEncrypt cert, since the LetsEncrypt Cert can only be generated for public DNS ?

        am I making sense? :)

        Thanks in advance

        Reply Report
    • davidpesticcio May 18, 2020

      TL;DR: You cannot use self-signed certificates for TLS with DOKS.

      It is totally possible - but it seems that you need to use a TLS server certificate on YOUR Atrifactory service that the DOKS nodes can validate - hence the link to LetsEncrypt website.

      Your kubernetes config which fetches the docker image, from wherever you like, via the kubernetes node. Which communicates with the docker engine on that DOKS node - which is configured to ONLY talk with docker registries that have TLS enabled, and ONLY with ones that use server certificates issued from a public CA.

      I’m making a presumption here - which seems pretty certain given the initial reply by @jkwiatkoski - that you cannot use a TLS server certificate on YOUR Artifactory service (or any other TLS connection) with a self-signed server certificate as the certificate cannot be validated by the DOKS node.

      This is because the DOKS nodes will NOT have the public key for YOUR server certificate, or intermediary, that is required for the DOKS node to validate YOUR self-signed certificate.

      It may help if you review some TLS certificate 101 material.

      Please see here for some basics;
      https://www.digitalocean.com/community/tutorials/a-comparison-of-let-s-encrypt-commercial-and-private-certificate-authorities-and-self-signed-ssl-certificates

      A Comparison of Let's Encrypt, Commercial and Private Certificate Authorities, and Self-Signed SSL Certificates
      by Brian Boucheron
      The push to get more and more web traffic secured with SSL encryption means that an increasing number of services and use-cases need a solution for obtaining the proper certificates. Whether it's a public website, intranet traffic, or a staging server for your web app, you'll...
      Reply Report
davidpesticcio May 18, 2020

Hi @MBII,

The reply from @jkwiatkoski suggests that all the nodes that make up the DOKS will only pull from a TLS enabled docker registry - I presume that specifically means that any DO docker registries are TLS enabled.

That being the case, you will need to enable TLS on your Artifactory service that is providing your docker registry.

You may have a “chicken & egg” scenario though if that Artifactory service is being deployed as a container to the DOKS, unless the initial image comes from DO docker registry! :-)

This issue may help you understand the cause of the problem, should you want to use an “insecure” (non-TLS enabled) registry elsewhere:

https://github.com/moby/moby/issues/28321

As mentioned though, you will need to use a TLC certificate that was issued by a public CA, rather than your private CA, and self-signed certificate.

You should find all that you need from the link mentioned by @jkwiatkoski though - failing that search the Digital Ocean Community pages for more help, or use your favourite search engine to discover a walkthrough! ;-)

If you can’t find the docs on Digital Ocean Community pages, and you discover a decent walkthrough - why not add it to the Digital Ocean Community! :-D

Good luck!

Reply Report

Related

Looking for something else?

Ask a new question Search for more help