We hope you find this tutorial helpful. In addition to guides like this one, we provide simple cloud infrastructure for developers. Learn more →

How To Troubleshoot SSH Protocol Issues On Your Droplet

PostedApril 28, 2017 1.6k views Linux Basics Arch Arch Linux CentOS CoreOS Debian FreeBSD RHEL Ubuntu Ubuntu 12.04 Ubuntu 16.04

Introduction

This is a continuation of our SSH troubleshooting article series.

The first article will help you determine when to troubleshoot an issue instead of migrating or redeploying, and provides resources for the information you'll need to have to troubleshoot effectively. The other parts of this series cover how to identify and resolve specific SSH errors, and are broken down by which phase of a successful SSH connection you need to debug:

Once a connection is successfully established, the SSH protocol negotiates an encrypted connection with the client based on establishing trust. Before you can even verify who you are on the system, the system will attempt to establish an encryption method with your client. There are a few unique issues that can occur in this scope, and this tutorial covers how to identify them, how to resolve them, and additional resources to prevent them.

Prerequisites

To troubleshoot SSH issues, you will need to make sure your Droplet is responding normally from the web console with a working network configuration. You can do this by following the How and When To Troubleshoot SSH Issues tutorial.

Before troubleshooting SSH, you should always check your cloud panel for ongoing issues in the region impacting your Droplet, the hypervisor status, and the state of the Droplet through the web console.

Errors

Below are some common SSH protocol errors you might encounter.

Host Key Verification Issues

When you connect to a server through an SSH client, the server attempts to identify itself using a host key. In situations where the host key changes, you may see a warning such as:

Warning output
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. ...

In PuTTY, you get an equally ominous warning:

PuTTY warning output
WARNING - POTENTIAL SECURITY BREACH! The server's host key does not match the one PuTTY has cached in the registry. This means that either the Server administrator has changed the host key, or you ...

Common causes for this issue include:

  • Rebuilding the Droplet from a snapshot or backup
  • Moving a Floating IP to another Droplet
  • Reinstalling the SSH server through the package manager

To resolve this, you can clear your host keys.

Connection Closed or Reset By Peer

In some situations, connectivity is established at the socket level but disconnects during the host key verification phase. In PuTTY you may see an error reported as:

PuTTY error output
Server Unexpectedly closed network connection

The OpenSSH client may report:

Error output
Connection closed by 111.111.111.111 port 22

Some common issues that may cause this error include:

  • The SSH service has crashed or is experiencing bugs.
  • The SSH service is not able to initialize a connection due missing server host keys.

In a situation like this, a more thorough review of the protocol output is necessary. In most cases, you will need to investigate this from the server through the web console. From there, you will want to make sure that the service is currently running and bound to the expected port.

If the service seems functional, next you should verify that host keys are present and regenerate them if they're not.

Unable to Negotiate with Host

When initiating the SSH protocol, a shared secret is generated through a cipher negotiated between the client and the host. When there is a mismatch, you may see errors in PuTTY like this:

PuTTY error output
Couldn't agree a client-to-server cipher (available: aes128-ctr, aes192-ctr, aes256-ctr)

An OpenSSH client may report:

Error output
Unable to negotiate with 111.111.111.111: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1

Some common issues that may cause this error include:

  • You have a modern OpenSSH client implementation and an older host that does not support a common cipher with the client.
  • You have customized your SSH client cipher list to only allow a selection that the server does not support.

To resolve this issue, you need to customize the supported ciphers in your SSH client.

Solutions

Below are some troubleshooting methods and solutions to common SSH protocol errors.

Clearing Out Host Keys from Known Hosts

Host keys are typically stored in the ~/.ssh/known_hosts file on OpenSSH client implementations. PuTTY typically stores these in the Windows Registry (HKEY_CURRENT_USER\Software\SimonTatham\PuTTY\SshHostKeys).

In PuTTY, you can use the dialogue box to select Yes to allow the connection and update the registry. Alternatively, you can manually remove the entry.

On an OpenSSH client, you can can find the host entry in the ~/.ssh/known_hosts file and manually remove it. Another option is to use the ssh-keygen command:

  • ssh-keygen -R your_server_ip

This will attempt to access and clear the matching host entry in the known_hosts file:

Output
# Host 111.111.111.111 found: line 2 /home/user/.ssh/known_hosts updated. Original contents retained as /home/user/.ssh/known_hosts.old

You can then try to reconnect to your server normally.

Checking and Regenerating SSH Host Keys

If your SSH host doesn't have its own private key to generate a shared secret, the connection will be aborted. To check that it does, look in the /etc/ssh directory for a group of files named like sshd_host_*_key with corresponding .pub files.

If they're not found, you'll need to regenerate them. You can do this using ssh-keygen as root or with sudo:

  • ssh-keygen -A

The output will report the actions taken, including a list of the keys generated:

Output
ssh-keygen: generating new host keys: RSA DSA ECDSA ED25519

You can then try to reconnect to your server normally.

Customizing Supported SSH Ciphers

You can customize the supported SSH ciphers on your client machine when you need support for a deprecated cipher like SHA1. This is not a very common issue. It typically happens in instances when you're using a newer SSH client to connect to an old SSH server that hasn't yet disabled weaker cyphers.

In OpenSSH, you can configure SHA1 using the KexAlgorithms option on the command line:

  • openssh -oKexAlgorithms=+diffie-hellman-group1-sha1 root@your_server_ip

PuTTY should already include the Diffie-Hellman group 1 option in the Connection > SSH > Kex configuration.

Conclusion

If you need further help establishing a successful protocol with your Droplet, you can open a ticket with our Support Team. Make sure to include:

  • The username, host, and port you are using to connect
  • The authentication mechanism you expect to use
  • The full output of the errors linked to the stage of error, including verbose output of the SSH client
  • All of the information you've gathered from troubleshooting so far
  • Anything you were unclear about while referencing this article

Including all the above diagnostic information and clarifying where you are encountering the issue when trying to connect can help us quickly get up to speed with where your need on the issue is.

0 Comments

Creative Commons License