Question

Django app for continuous deployment using Gitlab on DigitalOcean droplet return permission denied for SSH

Posted September 23, 2021 204 views
DockerDjangoCI/CDDigitalOcean Droplets

I’m trying to implement CD for my containerized(Docker, nginx) Django app using gitlab on DigitalOcean droplet.

I have created a pair of SSH keys and add the public key to DigitalOcean platform. I can login to my droplet using that SSH key.

Now, I have added the private key as the environment variable at gitlab as: $PRIVATE_KEY , so now when I run the deployment it return the permission denied error.

Here’s my : .gitlab-ci.yml:

image:
  name: docker/compose:1.29.2
  entrypoint: [""]

services:
  - docker:dind

stages:
  - build
  - deploy

variables:
  DOCKER_HOST: tcp://docker:2375
  DOCKER_DRIVER: overlay2

before_script:
  - export IMAGE=$CI_REGISTRY/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME
  - export WEB_IMAGE=$IMAGE/web:web
  - export NGINX_IMAGE=$IMAGE/nginx:nginx
  - apk add --no-cache openssh-client bash
  - chmod +x ./setup_env.sh
  - bash ./setup_env.sh
  - docker login -u $CI_REGISTRY_USER -p $CI_JOB_TOKEN $CI_REGISTRY

build:
  stage: build
  script:
    - docker pull $IMAGE/web:web || true
    - docker pull $IMAGE/nginx:nginx || true
    - docker-compose -f docker-compose.prod.yml build
    - docker push $IMAGE/web:web
    - docker push $IMAGE/nginx:nginx

deploy:
  stage: deploy
  script:
    - mkdir -p ~/.ssh
    - echo "$PRIVATE_KEY" | tr -d '\r' > ~/.ssh/id_ed25519
    - cat ~/.ssh/id_ed25519
    - chmod 700 ~/.ssh/id_ed25519
    - eval "$(ssh-agent -s)"
    - ssh-add ~/.ssh/id_ed25519
    - ssh-keyscan -H 'gitlab.com' >> ~/.ssh/known_hosts
    - chmod +x ./deploy.sh
    - scp  -o StrictHostKeyChecking=no -r ./.env ./docker-compose.prod.yml root@$DO_PUBLIC_IP_ADDRESS:/root/Pythonist.org
    - bash ./deploy.sh

The build stage is passed but the deploy is failed with the following error:

$ chmod +x ./setup_env.sh
$ bash ./setup_env.sh
$ docker login -u $CI_REGISTRY_USER -p $CI_JOB_TOKEN $CI_REGISTRY
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
$ mkdir -p ~/.ssh
$ echo "$PRIVATE_KEY" | tr -d '\r' > ~/.ssh/id_ed25519
$ cat ~/.ssh/id_ed25519
   <MY_PRIVATE_KEY>
$ chmod 700 ~/.ssh/id_ed25519
$ eval "$(ssh-agent -s)"
Agent pid 27
$ ssh-add ~/.ssh/id_ed25519
Identity added: /root/.ssh/id_ed25519 (<COMMENT>)
$ ssh-keyscan -H 'gitlab.com' >> ~/.ssh/known_hosts
# gitlab.com:22 SSH-2.0-OpenSSH_7.9p1 Debian-10+deb10u2
# gitlab.com:22 SSH-2.0-OpenSSH_7.9p1 Debian-10+deb10u2
# gitlab.com:22 SSH-2.0-OpenSSH_7.9p1 Debian-10+deb10u2
# gitlab.com:22 SSH-2.0-OpenSSH_7.9p1 Debian-10+deb10u2
# gitlab.com:22 SSH-2.0-OpenSSH_7.9p1 Debian-10+deb10u2
$ chmod +x ./deploy.sh
$ scp  -o StrictHostKeyChecking=no -r ./.env ./docker-compose.prod.yml root@$DO_PUBLIC_IP_ADDRESS:/root/app
Warning: Permanently added '143.198.103.99' (ECDSA) to the list of known hosts.
Permission denied, please try again.
Permission denied, please try again.
root@143.198.103.99: Permission denied (publickey,password).
lost connection
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1

Here’s my deploy.sh:

#!/bin/sh

ssh -o StrictHostKeyChecking=no root@$DO_PUBLIC_IP_ADDRESS << 'ENDSSH'
  cd /root/Pythonist.org
  export $(cat .env | xargs)
  docker login -u $CI_REGISTRY_USER -p $CI_JOB_TOKEN $CI_REGISTRY
  docker pull $IMAGE/web:web
  docker pull $IMAGE/nginx:nginx
  docker-compose -f docker-compose.prod.yml up -d
ENDSSH

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
Submit an Answer
1 answer

Hello,

What I could suggest trying is to add a new line at the end of the PRIVATE_KEY variable so that it matches the required format.

If this does not work, what I could suggest is adding the -vvv debug flag for the scp command so that you could see the actual error rather than just the permission denied message. This extra output might give you more information on why the connection is failing.

Let me know how it goes!
Regards,
Bobby

  • I have added -vvv in the scp command and here’s the output:

    $ scp  -o StrictHostKeyChecking=no -vvv -r ./.env ./docker-compose.prod.yml root@$DO_PUBLIC_IP_ADDRESS:/Pythonist.org
    Executing: program /usr/bin/ssh host <MY_IP>, user root, command scp -v -r -d -t /Pythonist.org
    OpenSSH_8.3p1, OpenSSL 1.1.1k  25 Mar 2021
    debug1: Reading configuration data /etc/ssh/ssh_config
    debug2: resolve_canonicalize: hostname <MY_IP> is address
    debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
    debug2: ssh_connect_direct
    debug1: Connecting to <MY_IP> [<MY_IP>] port 22.
    debug1: Connection established.
    load pubkey "/root/.ssh/id_rsa": invalid format
    debug1: identity file /root/.ssh/id_rsa type -1
    debug1: identity file /root/.ssh/id_rsa-cert type -1
    debug1: identity file /root/.ssh/id_dsa type -1
    debug1: identity file /root/.ssh/id_dsa-cert type -1
    debug1: identity file /root/.ssh/id_ecdsa type -1
    debug1: identity file /root/.ssh/id_ecdsa-cert type -1
    debug1: identity file /root/.ssh/id_ecdsa_sk type -1
    debug1: identity file /root/.ssh/id_ecdsa_sk-cert type -1
    debug1: identity file /root/.ssh/id_ed25519 type -1
    debug1: identity file /root/.ssh/id_ed25519-cert type -1
    debug1: identity file /root/.ssh/id_ed25519_sk type -1
    debug1: identity file /root/.ssh/id_ed25519_sk-cert type -1
    debug1: identity file /root/.ssh/id_xmss type -1
    debug1: identity file /root/.ssh/id_xmss-cert type -1
    debug1: Local version string SSH-2.0-OpenSSH_8.3
    debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6p1 Ubuntu-4ubuntu0.5
    debug1: match: OpenSSH_7.6p1 Ubuntu-4ubuntu0.5 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
    debug2: fd 3 setting O_NONBLOCK
    debug1: Authenticating to <MY_IP>:22 as 'root'
    debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
    debug3: send packet: type 20
    debug1: SSH2_MSG_KEXINIT sent
    debug3: receive packet: type 20
    debug1: SSH2_MSG_KEXINIT received
    debug2: local client KEXINIT proposal
    debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
    debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,ssh-ed25519,sk-ssh-ed25519@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
    debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
    debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
    debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
    debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
    debug2: compression ctos: none,zlib@openssh.com,zlib
    debug2: compression stoc: none,zlib@openssh.com,zlib
    debug2: languages ctos: 
    debug2: languages stoc: 
    debug2: first_kex_follows 0 
    debug2: reserved 0 
    debug2: peer server KEXINIT proposal
    debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
    debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
    debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
    debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
    debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
    debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
    debug2: compression ctos: none,zlib@openssh.com
    debug2: compression stoc: none,zlib@openssh.com
    debug2: languages ctos: 
    debug2: languages stoc: 
    debug2: first_kex_follows 0 
    debug2: reserved 0 
    debug1: kex: algorithm: curve25519-sha256
    debug1: kex: host key algorithm: ecdsa-sha2-nistp256
    debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
    debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
    debug3: send packet: type 30
    debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
    debug3: receive packet: type 31
    debug1: Server host key: ecdsa-sha2-nistp256 SHA256:RqoJriybF7in/WQV5FMycNzouGrBrj05n1UdGIsoWZ0
    debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
    Warning: Permanently added '<MY_IP>' (ECDSA) to the list of known hosts.
    debug3: send packet: type 21
    debug2: set_newkeys: mode 1
    debug1: rekey out after 134217728 blocks
    debug1: SSH2_MSG_NEWKEYS sent
    debug1: expecting SSH2_MSG_NEWKEYS
    debug3: receive packet: type 21
    debug1: SSH2_MSG_NEWKEYS received
    debug2: set_newkeys: mode 0
    debug1: rekey in after 134217728 blocks
    debug1: Will attempt key: /root/.ssh/id_rsa RSA SHA256:8P7SzdSfToYauwzXTVYdrkjuR7zhAg4fK5Z/BZFp9Aw agent
    debug1: Will attempt key: /root/.ssh/id_rsa 
    debug1: Will attempt key: /root/.ssh/id_dsa 
    debug1: Will attempt key: /root/.ssh/id_ecdsa 
    debug1: Will attempt key: /root/.ssh/id_ecdsa_sk 
    debug1: Will attempt key: /root/.ssh/id_ed25519 
    debug1: Will attempt key: /root/.ssh/id_ed25519_sk 
    debug1: Will attempt key: /root/.ssh/id_xmss 
    debug2: pubkey_prepare: done
    debug3: send packet: type 5
    debug3: receive packet: type 7
    debug1: SSH2_MSG_EXT_INFO received
    debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
    debug3: receive packet: type 6
    debug2: service_accept: ssh-userauth
    debug1: SSH2_MSG_SERVICE_ACCEPT received
    debug3: send packet: type 50
    debug3: receive packet: type 51
    debug1: Authentications that can continue: publickey,password
    debug3: start over, passed a different list publickey,password
    debug3: preferred publickey,keyboard-interactive,password
    debug3: authmethod_lookup publickey
    debug3: remaining preferred: keyboard-interactive,password
    debug3: authmethod_is_enabled publickey
    debug1: Next authentication method: publickey
    debug1: Offering public key: /root/.ssh/id_rsa RSA SHA256:8P7SzdSfToYauwzXTVYdrkjuR7zhAg4fK5Z/BZFp9Aw agent
    debug3: send packet: type 50
    debug2: we sent a publickey packet, wait for reply
    debug3: receive packet: type 51
    debug1: Authentications that can continue: publickey,password
    debug1: Trying private key: /root/.ssh/id_rsa
    debug3: sign_and_send_pubkey: RSA SHA256:8P7SzdSfToYauwzXTVYdrkjuR7zhAg4fK5Z/BZFp9Aw
    debug3: sign_and_send_pubkey: signing using rsa-sha2-512 SHA256:8P7SzdSfToYauwzXTVYdrkjuR7zhAg4fK5Z/BZFp9Aw
    debug3: send packet: type 50
    debug2: we sent a publickey packet, wait for reply
    debug3: receive packet: type 51
    debug1: Authentications that can continue: publickey,password
    debug1: Trying private key: /root/.ssh/id_dsa
    debug3: no such identity: /root/.ssh/id_dsa: No such file or directory
    debug1: Trying private key: /root/.ssh/id_ecdsa
    debug3: no such identity: /root/.ssh/id_ecdsa: No such file or directory
    debug1: Trying private key: /root/.ssh/id_ecdsa_sk
    debug3: no such identity: /root/.ssh/id_ecdsa_sk: No such file or directory
    debug1: Trying private key: /root/.ssh/id_ed25519
    debug3: no such identity: /root/.ssh/id_ed25519: No such file or directory
    debug1: Trying private key: /root/.ssh/id_ed25519_sk
    debug3: no such identity: /root/.ssh/id_ed25519_sk: No such file or directory
    debug1: Trying private key: /root/.ssh/id_xmss
    debug3: no such identity: /root/.ssh/id_xmss: No such file or directory
    debug2: we did not send a packet, disable method
    debug3: authmethod_lookup password
    debug3: remaining preferred: ,password
    debug3: authmethod_is_enabled password
    debug1: Next authentication method: password
    debug1: read_passphrase: can't open /dev/tty: No such device or address
    debug3: send packet: type 50
    debug2: we sent a password packet, wait for reply
    debug3: receive packet: type 51
    debug1: Authentications that can continue: publickey,password
    Permission denied, please try again.
    debug1: read_passphrase: can't open /dev/tty: No such device or address
    debug3: send packet: type 50
    debug2: we sent a password packet, wait for reply
    debug3: receive packet: type 51
    debug1: Authentications that can continue: publickey,password
    Permission denied, please try again.
    debug1: read_passphrase: can't open /dev/tty: No such device or address
    debug3: send packet: type 50
    debug2: we sent a password packet, wait for reply
    debug3: receive packet: type 51
    debug1: Authentications that can continue: publickey,password
    debug2: we did not send a packet, disable method
    debug1: No more authentication methods to try.
    root@<MY_IP>: Permission denied (publickey,password).
    lost connection
    
    • Hello,

      As far as I can see from the output, it looks like that for some reason the key is reported to be missing.

      I could suggest making sure that the .ssh directory is set to 700 and the key is set to 600:

          - mkdir -p ~/.ssh
          - chmod 700 ~/.ssh
          - echo "$PRIVATE_KEY" | tr -d '\r' > ~/.ssh/id_ed25519
          - cat ~/.ssh/id_ed25519
          - chmod 600 ~/.ssh/id_ed25519
      

      Let me know if this works, if not I just went through the GitLab documentation, and they recommend adding the SSH Key steps in the before_script stage:

      before_script:
      
        - eval $(ssh-agent -s)
      
        - echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
      
        - mkdir -p ~/.ssh
        - chmod 700 ~/.ssh
      

      Let me know how it goes!
      Best,
      Bobby

      • I have added that under before_script

        My current configuration is as:

        image:
          name: docker/compose:1.29.2
          entrypoint: [""]
        
        services:
          - docker:dind
        
        stages:
          - build
          - deploy
        
        variables:
          DOCKER_HOST: tcp://docker:2375
          DOCKER_DRIVER: overlay2
        
        before_script:
          - export IMAGE=$CI_REGISTRY/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME
          - export WEB_IMAGE=$IMAGE/web:web
          - export NGINX_IMAGE=$IMAGE/nginx:nginx
          - apk add --no-cache openssh-client bash
          - chmod +x ./setup_env.sh
          - bash ./setup_env.sh
          - docker login -u $CI_REGISTRY_USER -p $CI_JOB_TOKEN $CI_REGISTRY
          - mkdir -p ~/.ssh
          - chmod 700 ~/.ssh
          - echo "$PRIVATE_KEY" | tr -d '\r' > ~/.ssh/id_rsa
          - cat ~/.ssh/id_rsa
          - chmod 600 ~/.ssh/id_rsa
        
        build:
          stage: build
          script:
            - docker pull $IMAGE/web:web || true
            - docker pull $IMAGE/nginx:nginx || true
            - docker-compose -f docker-compose.prod.yml build
            - docker push $IMAGE/web:web
            - docker push $IMAGE/nginx:nginx
        
        deploy:
          stage: deploy
          script:
            - ssh-keyscan -H 'gitlab.com' >> ~/.ssh/known_hosts
            - chmod +x ./deploy.sh
            - scp  -o StrictHostKeyChecking=no -vvv -r ./.env ./docker-compose.prod.yml root@$DO_PUBLIC_IP_ADDRESS:/Pythonist.org
            - bash ./deploy.sh
        

        Still getting the same error.

        • Hello,

          This all looks correct. Currently the only thing that comes to my mind is that if you have a pass phrase for your SSH key, is this the case by any chance?

          If so, I would recommend generating an SSH key without a pass phrase and giving it another try.

          Let me know how it goes!
          Best,
          Bobby

          • I have created a new SSH key-pair named py_key and added the public key to DO and also copy it inside the ~/.ssh/authorized_keys. Then I have placed the private key named py_key inside the gitlab ENV variable named SSH_PRIVATE_KEY.

            And here’s the final configuration:

            image:
              name: docker/compose:1.29.2
              entrypoint: [""]
            
            services:
              - docker:dind
            
            stages:
              - build
              - deploy
            
            variables:
              DOCKER_HOST: tcp://docker:2375
              DOCKER_DRIVER: overlay2
            
            before_script:
              - export IMAGE=$CI_REGISTRY/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME
              - export WEB_IMAGE=$IMAGE/web:web
              - export NGINX_IMAGE=$IMAGE/nginx:nginx
              - apk add --no-cache openssh-client bash
              - chmod +x ./setup_env.sh
              - bash ./setup_env.sh
              - docker login -u $CI_REGISTRY_USER -p $CI_JOB_TOKEN $CI_REGISTRY
              - mkdir -p ~/.ssh
              - chmod 700 ~/.ssh
              - echo "$PRIVATE_KEY" | tr -d '\r' > ~/.ssh/py_key
              - cat ~/.ssh/py_key
              - chmod 600 ~/.ssh/py_key
            
            build:
              stage: build
              script:
                - docker pull $IMAGE/web:web || true
                - docker pull $IMAGE/nginx:nginx || true
                - docker-compose -f docker-compose.prod.yml build
                - docker push $IMAGE/web:web
                - docker push $IMAGE/nginx:nginx
            
            deploy:
              stage: deploy
              script:
                # - mkdir -p ~/.ssh
                # - echo "$SSH_PRIVATE_KEY" | tr -d '\r' > ~/.ssh/id_rsa
                - ssh-keyscan -H 'gitlab.com' >> ~/.ssh/known_hosts
                - chmod +x ./deploy.sh
                - scp  -o StrictHostKeyChecking=no -vvv -r ./.env ./docker-compose.prod.yml root@$DO_PUBLIC_IP_ADDRESS:/Pythonist.org
                - bash ./deploy.sh
            

            and here’s the logs output:
            https://marspaste.com/logs-of-gitlab-ciyml

          • Hello,

            Keep in mind that the naming of the key files matters, as your key is named py_key it is not being used during the SSH connection.

            There are two solutions here:

            • Change the name of the file to id_rsa
            • Specify the path to the key file in the scp command with the -i flag:
            scp -i ~/.ssh/py_key -o StrictHostKeyChecking=no -vvv -r ./.env ./docker-compose.prod.yml root@$DO_PUBLIC_IP_ADDRESS:/Pythonist.org
            

            Let me know how it goes.
            Best,
            Bobby

  • Hi @bobbyiliev I have tried that and it seems like the issue regarding SSH is resolved, but there’s another issue arises, while I was trying this one, I get out of minutes for gitlab shared runners, so I have added a new runner on DigitalOcean droplet and add that to the project, now When I triggered the build, it runs the build job successfully but on deploy, it failed with the following error:

    This job is stuck because the project doesn't have any runners online assigned to it.
    

    While I can see my runner which is active in CI settings of my project, and There’s no job running on this runner.

    What might happened?

    Kind regards,
    Abdul