Question

New images in a container registry cant be resolved.

Posted on September 1, 2024
DigitalOcean Container Registry (DOCR)DigitalOcean Managed Kubernetes
Asked by marlonmckinnie

I’ve got a kube cluster connected to a container registry having followed all of the guidance from the docs. I’ve been pushing images for different services for a while and I don’t believe I’ve done anything special outside of that.

Every time I push a new image to the same container registry as of late I get a failure to resolve the image citing :

  Warning  Failed     12m (x4 over 13m)     kubelet            Failed to pull image "registry.digitalocean.com/xxxxxx:xxxxxx": failed to pull and unpack image "registry.digitalocean.com/xxxxxx:xxxxxx": failed to resolve reference "registry.digitalocean.com/xxxxxx:xxxxxx": failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://api.digitalocean.com/v2/registry/auth?scope=repository%3Axxxxx%2Fxxx%2Fxxxx%3Apull&service=registry.digitalocean.com: 401 Unauthorized

It happened once before and last time I disconnected the cluster from the registry via the ui and reconnected it. I don’t think that worked initially but eventually, it started working again. I’m stuck in the same position again and I’m starting to wonder if there’s something about the way in pushing the initial image that could be causing this. when i push updates to existing images everything goes well. it seems to primarily be when I push the first version of an image. I’ve even tried updating the new image after that and pinning versions as I go but to no avail.

Submit an answer

This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

Bobby Iliev

Site Moderator

• September 3, 2024

Hey 👋

When you initially connect your Kubernetes cluster to DOCR, a secret is created that Kubernetes uses to authenticate and pull images. If this secret is missing, outdated, or not properly linked to the service accounts, it can cause the issue you’re seeing.

Verify the Secret: Run the following command to check if the secret exists and is properly configured:
```
kubectl get secrets
```
Look for a secret named something like registry-digitalocean-com, and ensure it’s linked to the correct service account.
Recreate the Secret: If the secret is missing or seems incorrect, you can recreate it using the following:
```
doctl kubernetes cluster registry add <cluster-name>
```
This command will automatically create or update the necessary secret in your Kubernetes cluster.

https://docs.digitalocean.com/reference/doctl/reference/kubernetes/cluster/registry/add/

Or you could manually recreate the secret as described here:

https://docs.digitalocean.com/products/container-registry/how-to/use-registry-docker-kubernetes/#create-secret-manually

As you mentioned, you’ve previously disconnected and reconnected the registry. This can sometimes refresh the authentication tokens and resolve the issue.

On another note, what you could do as a quick test is to try pulling the image manually on a local machine or within the cluster using docker pull to verify that the image is accessible:

docker pull registry.digitalocean.com/myregistry/myapp:v1.0.0

If this works locally but not on Kubernetes, the issue is likely with the cluster’s configuration.

If none of the above steps resolve the issue, it might be worth contacting DigitalOcean support, as there could be a backend issue with the registry service.