GitLab CI/CD Job fail Permission denied (publickey,password).

  • Posted February 18, 2020
  • CI/CD


I am trying to deploy my app from the git repository into my droplet.

I have generated a key pair, added it to the authorized_keys on the Droplet. Tested the ssh connection with gitlab via the ssh -T command.

Also I have encoded the privateKey in base64 and added it to the GitLab CI/CD Variables.

Using the following .gitlab-ci.yml

image: keymetrics/pm2:6

  - deploy

  stage: deploy
    - echo "====== Deploy to development server ======"
    - apk update && apk upgrade
    - apk add git openssh bash
    # Add target server`s secret key
    - mkdir ~/.ssh
    - echo $TARGET_SERVER_SECRET_KEY_BASE64_DEVELOPMENT | base64 -d > ~/.ssh/id_rsa
    - chmod 700 ~/.ssh && chmod 600 ~/.ssh/*
    - echo "Test ssh connection"
    - ssh -o StrictHostKeyChecking=no -T "ubuntu@$TARGET_SERVER_HOST_DEVELOPMENT"
    # Delploy
    - echo "Setup tagget server directories"
    - pm2 deploy ecosystem.config.js staging setup 2>&1 || true
    - echo "make deploy"
    - pm2 deploy ecosystem.config.js staging
    name: deploying
    - development

But I always get the following response

Test ssh connection
 $ ssh -o StrictHostKeyChecking=no -T "ubuntu@$TARGET_SERVER_HOST_DEVELOPMENT"
 Warning: Permanently added '' (ECDSA) to the list of known hosts.
 Permission denied, please try again.
 Permission denied, please try again.
 Permission denied (publickey,password).
 ERROR: Job failed: exit code 255

Is there something I am missing here?

Thanks in advance


Solved. The issue was the KEY variable, it was not accessible from the git CI script file

Submit an answer
You can type!ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

Hi @sabinpandelovitch

Let’s first being with the usual stuff, I know you mentioned you’ve actually checked the permissions and other stuff like that but let me post how they should be just in case:

  • Your home directory ~, your ~/.ssh directory and the ~/.ssh/authorized_keys file on the remote machine must be writable only by you: rwx------ and rwxr-xr-x are fine, but rwxrwx--- is no good, even if you are the only user in your group (if you prefer numeric modes: 700 or 755, not 775).
  • If ~/.ssh or authorized_keys is a symbolic link, the canonical path (with symbolic links expanded) is checked.
  • Your ~/.ssh/authorized_keys file (on the remote machine) must be readable (at least 400), but you’ll need it to be also writable (600) if you will add any more keys to it.
  • Your private key file (on the local machine) must be readable and writable only by you: rw-------, i.e. 600.

Now that we’ve passed the standard stuff, let’s get going on the more interesting stuff.

If you run the following command

/usr/sbin/sshd -d -p 2222

on your droplet, you should then connect without a problem. What concerns us is what does the debug information says on your droplet, It should state something like

Authentication allowed

In this case, what you can do is temporarily stop the SSH daemon and replace it with one in debug mode. Don’t worry, stopping the SSH daemon won’t kill any existing connections. This means it’s possible to run this without being connected to the droplet’s Console but it’s somewhat risky. If the connection does get broken for any kind of reason, you’ll need to connect using your droplet’s console. Anyway, you can run the following

service ssh stop
/usr/sbin/sshd -d
#...debug output...
service ssh start

If it again runs with the debug mode being on, then for sure it’s the SELinux causing the issues, it’s most probably set to Enforcing. The .ssh dir will probably be mislabeled. Look at /var/log/audit/audit.log. Check with ls -laZ and then Run restorecon -r -v /path/to/users/.ssh.

Regards, KDSys