SSH not working

April 19, 2017 7.5k views
DigitalOcean Ubuntu 16.04
ktottenATG
By:
ktottenATG

I'm having a problem using SSH to connect to any new droplets I have been spinning up. I am using a one-click app to create a Ubuntu Wordpress 4.7 on 16.04 droplet. When the droplet is first created, I can SSH using Mac's Terminal successfully. However, once I log out and try to log in either with Terminal, or to use Sequel Pro, or Transmit to SSH into the server, the connection is refused.

Using Terminal to try to SSH in, here is the error log:
OpenSSH6.9p1, LibreSSL 2.1.8
debug1: Reading configuration data /Users/ameerbacchus/.ssh/config
debug1: /Users/ameerbacchus/.ssh/config line 3: Applying options for *
debug1: Reading configuration data /etc/ssh/sshconfig
debug1: /etc/ssh/sshconfig line 21: Applying options for *
debug2: sshconnect: needpriv 0
debug1: Connecting to 138.197.154.119 [138.197.154.119] port 22.
debug1: connect to address 138.197.154.119 port 22: Connection refused
ssh: connect to host 138.197.154.119 port 22: Connection refused

Using Sequel Pro, I get this error message:

The SSH Tunnel could not authenticate with the remote host. Please check your password and ensure you still have access.

Used command: /usr/bin/ssh -v -N -S none -o ControlMaster=no -o ExitOnForwardFailure=yes -o ConnectTimeout=10 -o NumberOfPasswordPrompts=3 -o TCPKeepAlive=no -o ServerAliveInterval=60 -o ServerAliveCountMax=1 root @138.197.154.119 -L 52774:127.0.0.1:3306

OpenSSH6.9p1, LibreSSL 2.1.8
debug1: Reading configuration data /PATH/.ssh/config
debug1: /PATH/.ssh/config line 3: Applying options for *
debug1: Reading configuration data /etc/ssh/sshconfig
debug1: /etc/ssh/sshconfig line 21: Applying options for *
debug1: Control socket " none" does not exist
debug1: Connecting to 138.197.154.119 [138.197.154.119] port 22.
debug1: fd 3 clearing ONONBLOCK
debug1: Connection established.
debug1: identity file /PATH/.ssh/idrsa type 1
debug1: keyloadpublic: No such file or directory
debug1: identity file /PATH/.ssh/idrsa-cert type -1
debug1: keyloadpublic: No such file or directory
debug1: identity file /PATH/.ssh/iddsa type -1
debug1: keyloadpublic: No such file or directory
debug1: identity file /PATH/.ssh/iddsa-cert type -1
debug1: keyloadpublic: No such file or directory
debug1: identity file /PATH/.ssh/idecdsa type -1
debug1: keyloadpublic: No such file or directory
debug1: identity file /PATH/.ssh/idecdsa-cert type -1
debug1: keyloadpublic: No such file or directory
debug1: identity file /PATH/.ssh/ided25519 type -1
debug1: keyloadpublic: No such file or directory
debug1: identity file /PATH/.ssh/ided25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH6.9
debug1: Remote protocol version 2.0, remote software version OpenSSH7.2p2 Ubuntu-4ubuntu2.1
debug1: match: OpenSSH7.2p2 Ubuntu-4ubuntu2.1 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 138.197.154.119:22 as 'root '
debug1: SSH2MSGKEXINIT sent
debug1: SSH2MSGKEXINIT received
debug1: kex: server->client chacha20-poly1305@openssh.com <implicit> none
debug1: kex: client->server chacha20-poly1305@openssh.com <implicit> none
debug1: expecting SSH2MSGKEXECDHREPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:hl+ZkCGXBbWJUlKWyf2uWhLR3XcENlVSAi3TibfvDHU
Warning: Permanently added '138.197.154.119' (ECDSA) to the list of known hosts.
debug1: SSH2MSGNEWKEYS sent
debug1: expecting SSH2MSGNEWKEYS
debug1: SSH2MSGNEWKEYS received
debug1: SSH2MSGSERVICEREQUEST sent
debug1: SSH2MSGSERVICEACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /PATH/.ssh/idrsa
debug1: Authentications that can continue: publickey,password
debug1: Trying private key: /PATH/.ssh/iddsa
debug1: Trying private key: /PATH/.ssh/idecdsa
debug1: Trying private key: /PATH/.ssh/ided25519
debug1: Next authentication method: password
debug1: readpassphrase: can't open /dev/tty: Device not configured
debug1: permanentlydropsuid: 501
debug1: Authentications that can continue: publickey,password
Permission denied, please try again.
debug1: readpassphrase: can't open /dev/tty: Device not configured
debug1: permanentlydropsuid: 501
debug1: Authentications that can continue: publickey,password
Permission denied, please try again.
debug1: readpassphrase: can't open /dev/tty: Device not configured
debug1: permanentlydropsuid: 501
debug1: Authentications that can continue: publickey,password
debug1: No more authentication methods to try.
Permission denied (publickey,password).

Log In to Comment
3 Answers
jtittle1 April 19, 2017

@ktottenATG 

connect to address 138.197.154.119 port 22: Connection refused

Normally boils down to the firewall blocking your connection. This can happen for a few reasons, one of which is excessive login failures.

From what I recall, most of the one-click images for Ubuntu are setup with ufw, a firewall that acts as an overlay to iptables and simplifies the process of setting up most rule types.

If you're blocked, the only way to get back in is via Console which you can access from the DO CP.

Click on the name of the Droplet in question. From the left side menu, click Access. Now click on the button that says Launch Console.

You'll need your root password to login. If you only deployed with SSH Keys, the root password is disabled -- in such a case, you're effectively locked out of the server as the Console doesn't allow pubkey authentication.

If you do have a root password set, you'll be prompted to enter it in. You won't see anything when entering the password, so you'll need to type it in carefully. Once you have, hit enter.

Once logged in, you can run:

ufw disable

That'll turn ufw off. Keep in mind, it's not really recommended to run without a firewall active, so the best thing to do would be to simply reset the firewall and then setup new rules.

Once the firewall is disabled, try to login via SSH once again. If you're able to login, I would use the following to setup the firewall.

Double-Check ufw is disabled:

ufw disable

Reset the rules:

ufw reset

Setup Default Policies:

ufw default deny incoming && ufw default allow outgoing

We're setting the default policy to deny any incoming connection. This is preferred as we want to setup a deny all, except what we explicitly allow type policy. This is the first part.

Now, we'll add rules for SSH (Port 22), HTTP (Port 80), and HTTPS/SSL (Port 443). These are the ports we want to allow connections on. This ensures that only these three ports are open, no more, no less.

ufw allow 22/tcp && ufw allow 80/tcp && ufw allow 443/tcp

Now, I noticed that you're using Sequel Pro, which means we need to open another port, and that'd be 3306 as that's the port MySQL/MariaDB communicate on by default. The issue here is that with the base configuration for MySQL (which is what DigitalOcean uses), remote access to MySQL is not enabled, so you need to enable it.

You should have this file:

/etc/mysql/my.cnf

We need to open that file using nano and find bind-address. Normally this is set to 127.0.0.1, which is localhost. If you try to connect with bind-address set to that IP, it'll fail.

You need to change bind-address to either 0.0.0.0 or the IP address of your Droplet (the IPv4 IP) and then restart MySQL.

service mysql restart

To add MySQL to the firewall, we'd issue one more allow command, like so:

ufw allow 3306/tcp

With SSH, HTTP, HTTPS, and MySQL now added, we can enable ufw using:

ufw enable

It'll ask you to confirm -- do so -- and logout, then try to login again. Then try to login with Sequel Pro, etc.

...

Note: It's really not a good security practice to allow open access to port 3306 as is being done in the above guide. Ideally you would restrict access to a single IP, or set of IP's (if multiple people need access). Allowing open access, such as what is being done here allows anyone to attempt to login to MySQL on your server, whether the user(s) exist or not.

Reply
  • ktottenATG April 19, 2017

    jtittle- thank you for your response.

    "We need to open that file using nano and find bind-address. Normally this is set to 127.0.0.1, which is localhost. If you try to connect with bind-address set to that IP, it'll fail."

    -> This is odd, because I have set up 10 other installations the exact same way without an issue in the past. However, as of any new droplets I create, I seem to run into this issue.

    It appears the pattern is that I can log in via terminal and utilizing transmit without issue until I try to SSH utilizing Sequel Pro. Once I try to login with Sequel Pro and try to login with Terminal or Transmit, I am no longer able to SSH to the server. (I can utilize DO Console to log in to the server). However, it seemed as though if I came back to attempting to debugging this server later in the day, I could get back in with out issue until I tried using Sequel Pro again.

    When I disable ufw, the ability to login via SSH was not at first possible. After I reset ufw and added the allowances, and restarted apache2 and ssh for good measure, I was able to get back into the server.

    After changing the bind address, it does seem to have resolved this issue for the time being. Regarding:

    Note: It's really not a good security practice to allow open access to port 3306 as is being done in the above guide. Ideally you would restrict access to a single IP, or set of IP's (if multiple people need access). Allowing open access, such as what is being done here allows anyone to attempt to login to MySQL on your server, whether the user(s) exist or not.

    Would you mind passing on a link to any instructions whereby I might be able to restrict access to a set of IPs?

    • jtittle1 April 19, 2017

      @ktottenATG

      The block is most likely coming from the failed logins through Sequel Pro.

      The reasoning for this would be due to the fact that on the one-click images, port 3306 isn't open, thus if you attempt to connect to it, it'll fail. Multiple repeated attempts to connect to a port that isn't open will result in a block. The block will effectively block your IP entirely, not just your IP when connecting to that port, so once the block happens, you're only option is to connect using Console.

      The reason port 3306 isn't open is because on standard installations, it doesn't need to be since remote access in MySQL/MariaDB isn't enabled by default. As far as I remember, the configuration always binds to either 127.0.01 or localhost unless you're using a repository that doesn't use the standard configuration (and some don't).

      That being said, you can limit connections to a single IP or multiple IP's with ufw as well.

      To start, we need to remove the existing rule I provided above that allows open access to port 3306. To do that, we need to list the rules to see what number it is in the mix.

      ufw status numbered

      You should see something like this:

      Status: active

     To                         Action      From
     --                         ------      ----
[ 1] 22/tcp                     ALLOW IN    Anywhere                  
[ 2] 80/tcp                     ALLOW IN    Anywhere                  
[ 3] 443/tcp                    ALLOW IN    Anywhere                  
[ 4] 3306/tcp                   ALLOW IN    Anywhere                  
[ 5] 22/tcp (v6)                ALLOW IN    Anywhere (v6)             
[ 6] 80/tcp (v6)                ALLOW IN    Anywhere (v6)             
[ 7] 443/tcp (v6)               ALLOW IN    Anywhere (v6)             
[ 8] 3306/tcp (v6)              ALLOW IN    Anywhere (v6)

      The first rule we need to delete is #4, so we'll run:

      ufw delete 4

      Now if you run:

      ufw status numbered

      You should see the rule gone:

      Status: active

     To                         Action      From
     --                         ------      ----
[ 1] 22/tcp                     ALLOW IN    Anywhere                  
[ 2] 80/tcp                     ALLOW IN    Anywhere                  
[ 3] 443/tcp                    ALLOW IN    Anywhere                  
[ 4] 22/tcp (v6)                ALLOW IN    Anywhere (v6)             
[ 5] 80/tcp (v6)                ALLOW IN    Anywhere (v6)             
[ 6] 443/tcp (v6)               ALLOW IN    Anywhere (v6)             
[ 7] 3306/tcp (v6)              ALLOW IN    Anywhere (v6)

      Now we need to delete rule #7:

      ufw delete 7

      With the rules for port 3306 removed, we can now allow certain IP's through using:

      ufw allow from IPADDR to any port 3306

      You'll want to change IPADDR to your IP. So if your IP is 111.222.33.44, the command would look like:

      ufw allow from 111.222.33.44 to any port 3306

      You can repeat this for as many IP's as you'd like. Of course, if you don't have a static IP, you're going to be doing this often and blocks are going to happen if you forget to remove old IP's and add new ones.

      For that very reason, I normally setup a VPN on my Mac or Windows machine so that I have a static IP and then set the firewall rule to limit connections from the VPN IP since I know it's static and won't be changing unless I change it.

      Setting up a VPN can be a bit of a pain, so I normally recommend Algo:

      https://github.com/trailofbits/algo

      It makes setting up a working VPN simple as it handles everything for you. I use it on my Mac to deploy VPN's here at DigitalOcean. In about 5-10 minutes, I have a VPN that just works and doesn't require me to do the manual setup.

      • ktottenATG April 19, 2017

        I appreciate such detailed and informative responses- thank you much.

        However, I followed your instructions, but the problem is repeating itself and I am unable to SSH through Sequel Pro. I think I must be missing something, or there may be another configuration that is not set correctly.

        1. From a state in which I've reset apache, ssh, and sql and can successfully ssh into the server via terminal (with ufw enabled)- I attempt to login with Sequel Pro via SSH ( I tried using MySQL Host as both 127.0.0.1 and the IP address for the server), and it fails after one attempt. The setting in the config file has the bind address set to 0.0.0.0

        2. Next I try to SSH via terminal and my connection is refused.

        3. Going in through Digital Ocean Console, I set ufw to disabled. This should make it so I should be able to SSH via terminal immediately- correct? As it stands, it does not allow me to SSH via terminal.

        4. Try to telnet via port 22, connection refused.

        5. In the vain of the turn it off and turn it back on again method of fixing things I try the following: service apache2 restart, service ssh restart, ufw enable. SSH via terminal and telnet>connection refused.

        6. It only seems to let me SSH using terminal after some period of time passes, maybe 5-10 minutes.

        As it stands I am unable to repeat the earlier success I had in getting in with Sequel Pro, which may be in part because I lost the specific settings I had when I logged in, though I've tried every imaginable combo I could think of while ensuring my root password and sql password were correct.

        Please see error messages below:

        1. Seqel Pro Error ``` Used command: /usr/bin/ssh -v -N -S none -o ControlMaster=no -o ExitOnForwardFailure=yes -o ConnectTimeout=10 -o NumberOfPasswordPrompts=3 -o TCPKeepAlive=no -o ServerAliveInterval=60 -o ServerAliveCountMax=1 -p 22 root @138.197.154.119 -L 50219:127.0.0.1:3306

        OpenSSH6.9p1, LibreSSL 2.1.8
        debug1: Reading configuration data /Users/ameerbacchus/.ssh/config
        debug1: /Users/ameerbacchus/.ssh/config line 3: Applying options for *
        debug1: Reading configuration data /etc/ssh/ssh        config
        debug1: /etc/ssh/sshconfig line 21: Applying options for *
        debug1: Control socket " none" does not exist
        debug1: Connecting to 138.197.154.119 [138.197.154.119] port 22.
        debug1: fd 3 clearing O        NONBLOCK
        debug1: Connection established.
        debug1: identity file /Users/ameerbacchus/.ssh/idrsa type 1
        debug1: key        loadpublic: No such file or directory
        debug1: identity file /Users/ameerbacchus/.ssh/id        rsa-cert type -1
        debug1: keyloadpublic: No such file or directory
        debug1: identity file /Users/ameerbacchus/.ssh/iddsa type -1
        debug1: key        loadpublic: No such file or directory
        debug1: identity file /Users/ameerbacchus/.ssh/id        dsa-cert type -1
        debug1: keyloadpublic: No such file or directory
        debug1: identity file /Users/ameerbacchus/.ssh/idecdsa type -1
        debug1: key        loadpublic: No such file or directory
        debug1: identity file /Users/ameerbacchus/.ssh/id        ecdsa-cert type -1
        debug1: keyloadpublic: No such file or directory
        debug1: identity file /Users/ameerbacchus/.ssh/ided25519 type -1
        debug1: key        loadpublic: No such file or directory
        debug1: identity file /Users/ameerbacchus/.ssh/id        ed25519-cert type -1
        debug1: Enabling compatibility mode for protocol 2.0
        debug1: Local version string SSH-2.0-OpenSSH6.9
        debug1: Remote protocol version 2.0, remote software version OpenSSH        7.2p2 Ubuntu-4ubuntu2.1
        debug1: match: OpenSSH7.2p2 Ubuntu-4ubuntu2.1 pat OpenSSH* compat 0x04000000
        debug1: Authenticating to 138.197.154.119:22 as 'root '
        debug1: SSH2        MSGKEXINIT sent
        debug1: SSH2        MSGKEXINIT received
        debug1: kex: server->client chacha20-poly1305@openssh.com <implicit> none
        debug1: kex: client->server chacha20-poly1305@openssh.com <implicit> none
        debug1: expecting SSH2        MSGKEXECDHREPLY
        debug1: Server host key: ecdsa-sha2-nistp256 SHA256:hl+ZkCGXBbWJUlKWyf2uWhLR3XcENlVSAi3TibfvDHU
        Warning: Permanently added '138.197.154.119' (ECDSA) to the list of known hosts.
        debug1: SSH2        MSGNEWKEYS sent
        debug1: expecting SSH2        MSGNEWKEYS
        debug1: SSH2        MSGNEWKEYS received
        debug1: SSH2        MSGSERVICEREQUEST sent
        debug1: SSH2MSGSERVICEACCEPT received
        debug1: Authentications that can continue: publickey,password
        debug1: Next authentication method: publickey
        debug1: Offering RSA public key: /Users/ameerbacchus/.ssh/id        rsa
        debug1: Authentications that can continue: publickey,password
        debug1: Trying private key: /Users/ameerbacchus/.ssh/iddsa
        debug1: Trying private key: /Users/ameerbacchus/.ssh/id        ecdsa
        debug1: Trying private key: /Users/ameerbacchus/.ssh/ided25519
        debug1: Next authentication method: password
        debug1: read        passphrase: can't open /dev/tty: Device not configured
        debug1: permanentlydropsuid: 501
        debug1: Authentications that can continue: publickey,password
        Permission denied, please try again.
        debug1: readpassphrase: can't open /dev/tty: Device not configured
        debug1: permanently        dropsuid: 501
        debug1: Authentications that can continue: publickey,password
        Permission denied, please try again.
        debug1: read        passphrase: can't open /dev/tty: Device not configured
        debug1: permanentlydropsuid: 501
        debug1: Authentications that can continue: publickey,password
        debug1: No more authentication methods to try.
        Permission denied (publickey,password).

        2. Terminal SSH Error

        Ameers-MacBook-Pro:~ ameerbacchus$ ssh -vvv root@138.197.154.119
        OpenSSH6.9p1, LibreSSL 2.1.8
        debug1: Reading configuration data /Users/ameerbacchus/.ssh/config
        debug1: /Users/ameerbacchus/.ssh/config line 3: Applying options for *
        debug1: Reading configuration data /etc/ssh/ssh        config
        debug1: /etc/ssh/sshconfig line 21: Applying options for *
        debug2: ssh        connect: needpriv 0
        debug1: Connecting to 138.197.154.119 [138.197.154.119] port 22.
        debug1: connect to address 138.197.154.119 port 22: Connection refused
        ssh: connect to host 138.197.154.119 port 22: Connection refused

MarkyMark April 19, 2017

The two logs (terminal and sequence-pro) look quite different to me.

The connection refused message by the destination in the Terminal attempt indicates that SSH isn't even running (or configured for a different port) on port 22. I.e. nothing is serving that port.

With the other you are getting a connection (i.e. the ssh server answers), but the authentication fails.

What do you see if you telnet to the machine on port 22? 

telnet 138.197.154.119 22

Ideally you should see a prompt saying OpenSSH and the version number. You can't do more with Telnet but it indicates that the server is at least up and answering.

Reply
jtittle1 April 19, 2017

@ktottenATG

I just installed Sequel Pro back on my MBP and setup a stock installation of MariaDB (MySQL) using a bind address of 0.0.0.0 on a demo Droplet.

So bind-address looks like bind-address = 0.0.0.0.

I setup the same ufw rules that I provided you with:

ufw default deny incoming \
&& ufw default allow outgoing \
&& ufw allow 22/tcp \
&& ufw allow 80/tcp \
&& ufw allow 443/tcp

I then added the IP-Specific rule to only allow my IP to connect on port 3306.

ufw allow from MY_IP to any port 3306

Where MY_IP is my IP address (that of my VPN, so that I have a static IP).

When connecting to MySQL through Sequel Pro, my configuration looks like this:

http://oi68.tinypic.com/309seop.jpg

When testing the configuration, I get a success message. When trying to log in and out via SSH and using Sequel Pro over SSH, I'm not seeing any blocks occur, nor denial of access. I am using an SSH Key to login, though it prompts me for the key passphrase as expected and logs me in without any issues.

...

If you're not using Sequel Pro via SSH, and instead using Sequel Pro's Standard Connection method, you will need to add another user.

For example, if I'm connecting from my VPN IP, it's not allowed to connect, so I need to give access to my IP by creating a second user.

grant all on dbname.* to 'dbuser'@'MYIP' identified by 'mypass';

Where MYIP is my VPN IP. So I end up with two users.

'dbuser'@'localhost'

and

'dbuser'@'MYIP'
Reply
  • ktottenATG April 20, 2017

    @jtittle , utilizing an sshkey rather than just using root login and password seems to resolve the issue. I was trying to avoid having to use sshkeys, to try to reduce the complexity involved for others who need to access the server. But, I suppose it's better to err on the side of caution.

    I didn't actually have to go in and open the ports, change the ufw firewall settings or change the sql configuration file on the one-click install of Wordpress 4.7 on Ubuntu 16.04.

    It also seems like there is a plugin that comes in the default one-click image called fail2ban that also adds rules to the iptables on top of ufw.

    Thank you very much for your assistance.

  • emyrold January 4, 2019

    I have a default install of LEMP one-click 18.4 . as soon as I edit the my.cnf to add bind-address = 0.0.0.0 and then systemctl restart mysql it fails to start.... This seems like the most comprehensive howto I have found so far. I . have been at this for over 4 hours now.

Have another answer? Share your knowledge.

Related Questions