Question

Pushing images to DigitalOcean from Github Actions sometimes super slow and then fail due to auth issues

I am using Github Actions to build and push images into the DO registry container. I have 5 registries/5GB plan. 1-2 out of 5 times all work well, but then when I build and push a new image I can see some images pushed, but some retrying multiple times and then fail with the message: unauthorized: authentication required

Restarting Github Action will eventually successfully deploy, but could take 3-4 restarts sometimes.

Here is a full log of that step:

Run docker push $(echo $REGISTRY)/$(echo $IMAGE_NAME):$(echo $GITHUB_SHA | head -c7)
  shell: /usr/bin/bash -e {0}
  env:
    REGISTRY: registry.digitalocean.com/activity-forecast-registry
    IMAGE_NAME: maf-app-dev
The push refers to repository [registry.digitalocean.com/activity-forecast-registry/maf-app-dev]
5e04cb2e9305: Preparing
0d2f700afc0b: Preparing
7d8e1eca32de: Preparing
02951c770be7: Preparing
bdade195d586: Preparing
c33601de6b1b: Preparing
afc702ae7d27: Preparing
9b949d6ba996: Preparing
c3b4a43df7d5: Preparing
3cddc64f59e2: Preparing
b951f8a113f5: Preparing
1d8bcb7a961e: Preparing
f1417ff83b31: Preparing
9b949d6ba996: Waiting
c3b4a43df7d5: Waiting
3cddc64f59e2: Waiting
b951f8a113f5: Waiting
1d8bcb7a961e: Waiting
f1417ff83b31: Waiting
c33601de6b1b: Waiting
afc702ae7d27: Waiting
02951c770be7: Pushed
bdade195d586: Pushed
c33601de6b1b: Retrying in 5 seconds
c33601de6b1b: Retrying in 4 seconds
c33601de6b1b: Retrying in 3 seconds
c33601de6b1b: Retrying in 2 seconds
c33601de6b1b: Retrying in 1 second
afc702ae7d27: Retrying in 5 seconds
afc702ae7d27: Retrying in 4 seconds
afc702ae7d27: Retrying in 3 seconds
afc702ae7d27: Retrying in 2 seconds
afc702ae7d27: Retrying in 1 second
afc702ae7d27: Pushed
c33601de6b1b: Pushed
9b949d6ba996: Pushed
3cddc64f59e2: Layer already exists
b951f8a113f5: Layer already exists
1d8bcb7a961e: Layer already exists
f1417ff83b31: Layer already exists
0d2f700afc0b: Retrying in 5 seconds
0d2f700afc0b: Retrying in 4 seconds
0d2f700afc0b: Retrying in 3 seconds
0d2f700afc0b: Retrying in 2 seconds
0d2f700afc0b: Retrying in 1 second
5e04cb2e9305: Pushed
0d2f700afc0b: Pushed
unauthorized: authentication required
Error: Process completed with exit code 1.

Here is what I have in the Github Actions yml:

 - name: Build container image
        run: docker build -t $(echo $REGISTRY)/$(echo $IMAGE_NAME):$(echo $GITHUB_SHA | head -c7) .

      - name: Install doctl
        uses: digitalocean/action-doctl@v2
        with:
          token: ${{ secrets.DIGITALOCEAN_ACCESS_TOKEN }}

      - name: Log in to DigitalOcean Container Registry with short-lived credentials
        run: doctl registry login --expiry-seconds 600

      - name: Remove all old images
        run: if [ ! -z "$(doctl registry repository list | grep "$(echo $IMAGE_NAME)")" ]; then doctl registry repository delete-manifest $(echo $IMAGE_NAME) $(doctl registry repository list-tags $(echo $IMAGE_NAME) | grep -o "sha.*") --force; else echo "No repository"; fi

      - name: Push image to DigitalOcean Container Registry
        run: docker push $(echo $REGISTRY)/$(echo $IMAGE_NAME):$(echo $GITHUB_SHA | head -c7)

      - name: Run registry garbage collection
        run: doctl registry garbage-collection start --include-untagged-manifests --force

after that, I run the deployment.

Show comments

Submit an answer


This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

Sign In or Sign Up to Answer

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

alexdo
Site Moderator
Site Moderator badge
May 21, 2023
Accepted Answer

Hello @giantelectricblueshrimp

Note that Docker limits and retries a certain number of concurrent image uploads, which might affect your workflow.

By default, the Docker daemon will push five layers of an image at a time. If you are on a low bandwidth connection this may cause timeout issues and you may want to lower this via the --max-concurrent-uploads daemon option. See the daemon documentation for more details.

If the issue persists you can reach out to our support team as well.

Hope that this helps!

Recently, we have been encountering the same issues with our CI/CD process. To resolve this, we decided to relocate our container registry from SFO to AMS and it worked!

@alexdo I think that solved it! here is GitHub Action to set it if some one is having the same problem:

jobs:
  build_and_push:
    runs-on: ubuntu-latest
    steps:
      - name: Update Docker settings
        run: |
          sudo sed -i 's/ }/, \"max-concurrent-uploads\": 2 }/' /etc/docker/daemon.json
          sudo systemctl restart docker

Try DigitalOcean for free

Click below to sign up and get $200 of credit to try our products over 60 days!

Sign up

Get our biweekly newsletter

Sign up for Infrastructure as a Newsletter.

Hollie's Hub for Good

Working on improving health and education, reducing inequality, and spurring economic growth? We'd like to help.

Become a contributor

Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.

Welcome to the developer cloud

DigitalOcean makes it simple to launch in the cloud and scale up as you grow — whether you're running one virtual machine or ten thousand.

Learn more
DigitalOcean Cloud Control Panel