Question

GitLab CI/CD Job fail Permission denied (publickey,password).

Posted February 18, 2020 440 views
CI/CD

Hello,

I am trying to deploy my app from the git repository into my droplet.

I have generated a key pair, added it to the authorized_keys on the Droplet. Tested the ssh connection with gitlab via the ssh -T git@gitlab.com command.

Also I have encoded the privateKey in base64 and added it to the GitLab CI/CD Variables.

Using the following .gitlab-ci.yml

image: keymetrics/pm2:6

stages:
  - deploy

deploy_staging:
  stage: deploy
  script:
    - echo "====== Deploy to development server ======"
    - apk update && apk upgrade
    - apk add git openssh bash
    # Add target server`s secret key
    - mkdir ~/.ssh
    - echo $TARGET_SERVER_SECRET_KEY_BASE64_DEVELOPMENT | base64 -d > ~/.ssh/id_rsa
    - chmod 700 ~/.ssh && chmod 600 ~/.ssh/*
    - echo "Test ssh connection"
    - ssh -o StrictHostKeyChecking=no -T "ubuntu@$TARGET_SERVER_HOST_DEVELOPMENT"
    # Delploy
    - echo "Setup tagget server directories"
    - pm2 deploy ecosystem.config.js staging setup 2>&1 || true
    - echo "make deploy"
    - pm2 deploy ecosystem.config.js staging
  environment:
    name: deploying
  only:
    - development

But I always get the following response

Test ssh connection
 $ ssh -o StrictHostKeyChecking=no -T "ubuntu@$TARGET_SERVER_HOST_DEVELOPMENT"
 Warning: Permanently added '104.248.80.132' (ECDSA) to the list of known hosts.
 Permission denied, please try again.
 Permission denied, please try again.
 Permission denied (publickey,password).
 ERROR: Job failed: exit code 255

Is there something I am missing here?

Thanks in advance

Add a comment
sabinpandelovitch
By:
sabinpandelovitch
Add a comment
1 comment

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
1 answer
KDSys February 18, 2020

Hi @sabinpandelovitch

Let’s first being with the usual stuff, I know you mentioned you’ve actually checked the permissions and other stuff like that but let me post how they should be just in case:

  • Your home directory ~, your ~/.ssh directory and the ~/.ssh/authorized_keys file on the remote machine must be writable only by you: rwx------ andrwxr-xr-x are fine, but rwxrwx---is no good, even if you are the only user in your group (if you prefer numeric modes: 700 or 755, not 775).
  • If~/.ssh or authorized_keys is a symbolic link, the canonical path (with symbolic links expanded) is checked.
  • Your ~/.ssh/authorized_keys file (on the remote machine) must be readable (at least 400), but you’ll need it to be also writable (600) if you will add any more keys to it.
  • Your private key file (on the local machine) must be readable and writable only by you: rw——-, i.e. 600.

Now that we’ve passed the standard stuff, let’s get going on the more interesting stuff.

If you run the following command 

/usr/sbin/sshd -d -p 2222

on your droplet, you should then connect without a problem. What concerns us is what does the debug information says on your droplet, It should state something like 

Authentication allowed

In this case, what you can do is temporarily stop the SSH daemon and replace it with one in debug mode. Don’t worry, stopping the SSH daemon won’t kill any existing connections. This means it’s possible to run this without being connected to the droplet’s Console but it’s somewhat risky. If the connection does get broken for any kind of reason, you’ll need to connect using your droplet’s console. Anyway, you can run the following 

service ssh stop
/usr/sbin/sshd -d
#...debug output...
service ssh start

If it again runs with the debug mode being on, then for sure it’s the SELinux causing the issues, it’s most probably set to Enforcing. The .ssh dir will probably be mislabeled. Look at /var/log/audit/audit.log. Check with ls -laZ and then Run restorecon -r -v /path/to/users/.ssh.

Regards,
KDSys

Reply Report
  • KDSys February 18, 2020

    One other thing I just though of is actually testing the connection from outside with the verbose option of SSH on.That way you can see exactly what’s going on.

    Regards,
    KDSys

    Reply Report
Submit an Answer

Related

Looking for something else?

Ask a new question Search for more help