Question

Gitlab CI/CD Error

Posted on August 13, 2024
CI/CD
Asked by Raj Solanki

Hi, I am trying to run a gitlab CI/CD pipeline job and I am coming across the following errors in the publish stage. My VPS is running on Ubuntu LTS 22.04 and I’ve created a separate user called deployer that has been added to the docker group. I’ve checked the gitlab-runner service and it’s running as well as the runner I setup.

This is what I used to setup my gitlab runner for this repo

 sudo gitlab-runner register \
  --non-interactive \
  --url "https://gitlab.com" \
  --token "<token>" \
  --executor "docker" \
  --docker-image "docker:stable" \
  --description "Deployment Runner"

I’ve never come across this error before and not sure what it is. It looks like a permissions issue or something wrong with docker-dind.

Running with gitlab-runner 17.2.1 (9882d9c7)
  on Deployment Runner ezRnxw45L, system ID: s_1988bd8ca31d
Preparing the "docker" executor00:36
Using Docker executor with image docker:latest ...
Starting service docker:dind...
Pulling docker image docker:dind ...
Using docker image sha256:60ded0dc5246d628d5d5a71f9594efe5ad4752492fd677569cc163b43a5d0ffa for docker:dind with digest docker@sha256:a690693976550aba640859bb3c3c29eb323a4f53f684c99b2a8282b14a22308b ...
Waiting for services to be up and running (timeout 30 seconds)...
*** WARNING: Service runner-ezrnxw45l-project-57789842-concurrent-0-4f29f73ca82d2b5b-docker-0 probably didn't start properly.
Health check error:
service "runner-ezrnxw45l-project-57789842-concurrent-0-4f29f73ca82d2b5b-docker-0-wait-for-service" timeout
Health check container logs:
2024-08-13T22:18:31.412481653Z waiting for TCP connection to 172.17.0.2 on [2375 2376]...
2024-08-13T22:18:31.419879752Z dialing 172.17.0.2:2375...
2024-08-13T22:18:31.426979610Z dialing 172.17.0.2:2376...
2024-08-13T22:18:32.427448560Z dialing 172.17.0.2:2375...
2024-08-13T22:18:32.427481860Z dialing 172.17.0.2:2376...
2024-08-13T22:18:33.427782497Z dialing 172.17.0.2:2375...
2024-08-13T22:18:33.427848838Z dialing 172.17.0.2:2376...
Service container logs:
2024-08-13T22:18:32.251005676Z Certificate request self-signature ok
2024-08-13T22:18:32.252740734Z subject=CN=docker:dind server
2024-08-13T22:18:32.279933821Z /certs/server/cert.pem: OK
2024-08-13T22:18:33.134109853Z Certificate request self-signature ok
2024-08-13T22:18:33.135453444Z subject=CN=docker:dind client
2024-08-13T22:18:33.156984245Z /certs/client/cert.pem: OK
2024-08-13T22:18:33.161216959Z cat: can't open '/proc/net/ip6_tables_names': No such file or directory
2024-08-13T22:18:33.162129607Z cat: can't open '/proc/net/arp_tables_names': No such file or directory
2024-08-13T22:18:33.165349497Z ip: can't find device 'nf_tables'
2024-08-13T22:18:33.167425697Z nf_tables             266240 55 nft_chain_nat,nft_counter,nft_compat
2024-08-13T22:18:33.168217467Z nfnetlink              20480 4 nf_conntrack_netlink,nft_compat,nf_tables
2024-08-13T22:18:33.168245937Z libcrc32c              16384 5 nf_nat,nf_conntrack,nf_tables,btrfs,raid456
2024-08-13T22:18:33.169140212Z modprobe: can't change directory to '/lib/modules': No such file or directory
2024-08-13T22:18:33.172163806Z ip: can't find device 'ip_tables'
2024-08-13T22:18:33.174323742Z ip_tables              32768 0
2024-08-13T22:18:33.174856540Z x_tables               53248 5 xt_conntrack,xt_MASQUERADE,xt_addrtype,nft_compat,ip_tables
2024-08-13T22:18:33.175779289Z modprobe: can't change directory to '/lib/modules': No such file or directory
2024-08-13T22:18:33.177340019Z ip: can't find device 'ip6_tables'
2024-08-13T22:18:33.180182018Z modprobe: can't change directory to '/lib/modules': No such file or directory
2024-08-13T22:18:33.182473011Z iptables v1.8.10 (nf_tables)
2024-08-13T22:18:33.185721624Z mount: permission denied (are you root?)
2024-08-13T22:18:33.185999392Z Could not mount /sys/kernel/security.
2024-08-13T22:18:33.186150635Z AppArmor detection and --privileged mode might break.
2024-08-13T22:18:33.187802512Z mount: permission denied (are you root?)
*********
Pulling docker image docker:latest ...
Using docker image sha256:60ded0dc5246d628d5d5a71f9594efe5ad4752492fd677569cc163b43a5d0ffa for docker:latest with digest docker@sha256:a690693976550aba640859bb3c3c29eb323a4f53f684c99b2a8282b14a22308b ...
Preparing environment00:01
Running on runner-ezrnxw45l-project-57789842-concurrent-0 via lawdocs...
Getting source from Git repository00:01
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /builds/dev-top-tech/lawdocs-nextjs/.git/
Checking out 1694ffed as detached HEAD (ref is master)...
Skipping Git submodules setup
Executing "step_script" stage of the job script00:01
Using docker image sha256:60ded0dc5246d628d5d5a71f9594efe5ad4752492fd677569cc163b43a5d0ffa for docker:latest with digest docker@sha256:a690693976550aba640859bb3c3c29eb323a4f53f684c99b2a8282b14a22308b ...
$ docker build -t $TAG_COMMIT-landing -t $TAG_LATEST-landing -f landing/Dockerfile ./landing
ERROR: error during connect: Head "[http://docker:2375/_ping](http://docker:2375/_ping)": dial tcp: lookup docker on 67.207.67.3:53: no such host
Cleaning up project directory and file based variables 00:01
ERROR: Job failed: exit code 1

This is my .gitlab-ci.yml file:

 stages:
  - publish
  # - deploy
  # - cleanup

variables:
  TAG_LATEST: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_NAME:latest
  TAG_COMMIT: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_NAME:$CI_COMMIT_SHORT_SHA

publish:
  image: docker:latest
  stage: publish
  services:
    - docker:dind
  script:
    - docker build -t $TAG_COMMIT-landing -t $TAG_LATEST-landing -f landing/Dockerfile ./landing
    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
    - docker push $TAG_COMMIT-landing
    - docker push $TAG_LATEST-landing
deploy:
  image: alpine:latest
  stage: deploy
  before_script:
    - chmod og= $ID_RSA
    - apk update && apk add openssh-client
    - PROJECT_FOLDER=$(echo $TAG_COMMIT | cut -d'/' -f2 | cut -d':' -f1)
  script:
    - |
      ssh -i $ID_RSA -o StrictHostKeyChecking=no $SERVER_USER@$SERVER_IP "
        cd /home/deployer && \
        rm -rf temp_repo && \
        git clone $CI_REPOSITORY_URL temp_repo && \
        cd temp_repo && \
        cp docker-compose.yml docker-compose.prod.yml /home/deployer/test/ && \
        cd .. && rm -rf temp_repo && \
        cd /home/deployer/test && \
        docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY && \
        docker pull $TAG_LATEST-landing && \
        docker compose down || true && \
        docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --force-recreate
      "
  environment:
    name: production
    url: http://$SERVER_IP
  only:
    - master

cleanup:
  stage: cleanup
  before_script:
    - chmod og= $ID_RSA
    - apk update && apk add openssh-client
  script:
    - |
      ssh -i $ID_RSA -o StrictHostKeyChecking=no $SERVER_USER@$SERVER_IP "
        docker image prune -af
      "
  only:
    - master

Any help would be greatly appreciated it.

Submit an answer

This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

Bobby Iliev

Site Moderator

• August 25, 2024

Accepted Answer

Hey!

I think that I’ve seen this in the past with the docker:dind service as it requires the GitLab Runner to run in privileged mode to function correctly.

As far as I remember this was necessary because Docker-in-Docker needs more permissions than usual to operate. So what you could do is in your GitLab Runner configuration to specify it to run in privileged mode by adding the --privileged flag during the runner registration or editing the runner’s configuration file:

sudo gitlab-runner register \
  --non-interactive \
  --url "https://gitlab.com" \
  --token "<token>" \
  --executor "docker" \
  --docker-image "docker:stable" \
  --description "Deployment Runner" \
  --docker-privileged