How to Set Up SSH Keys on Ubuntu: A Comprehensive Guide

Secure Shell, or SSH, is the industry-standard protocol for securely administering remote systems. When you connect to a server, the first and most critical step is authentication. While passwords are a common method, they are susceptible to modern brute-force attacks, which can compromise your server’s security. To establish a more robust and reliable defense, you should use SSH keys. This method relies on public-key cryptography to provide a highly secure and convenient way to access your server, forming a foundational security practice for any system administrator.

This tutorial provides a step-by-step guide to setting up and using SSH keys on an Ubuntu server. You will learn how to generate a new, secure key pair on your local machine, correctly copy your public key to the server for authentication, and manage permissions. Finally, we will cover the essential hardening step of disabling password-based authentication on your server, ensuring that only users with a valid private key can gain access. By the end of this guide, you will have a secure, passwordless login configured for your server.

Secure Shell (SSH) is a fundamental protocol for secure remote system administration. While many users start by authenticating with passwords, SSH keys offer a more secure, convenient, and flexible method of authentication. This section covers the core concepts of SSH keys, why you should use them, and the different types available.

What Are SSH Keys?

SSH keys are a pair of cryptographic keys that can be used to authenticate to an SSH server as an alternative to password-based logins. This authentication method relies on asymmetric cryptography, also known as public-key cryptography.

Here’s how it works:

1. Key Pair Generation: You generate a pair of keys on your local machine: a private key and a public key.
2. The Private Key: The private key must be kept secret and secure. It resides on the client machine you use to connect to the server. Anyone who has your private key can impersonate you, so it’s critical to protect it with a strong passphrase.
3. The Public Key: The public key can be shared freely. To enable key-based authentication, you copy your public key to the remote server and add it to a special file in your user account, typically ~/.ssh/authorized_keys.

When you attempt to connect to the server, the server uses your public key to create a challenge message and sends it to your client. Your client then uses your private key to correctly respond to the challenge, proving your identity without ever transmitting the key itself. If the response is valid, the server grants you access.

Why Use SSH Keys?

Using SSH keys instead of passwords provides significant advantages in security, convenience, and automation.

• Enhanced Security: SSH keys are much more difficult to compromise than passwords. A typical password might be 8-16 characters long, drawing from a limited set of characters. An SSH key, on the other hand, is a long string of bits. A 256-bit Ed25519 key or a 4096-bit RSA key is computationally infeasible to crack using brute-force methods. This makes your server significantly more resilient to automated attacks.

• Convenience: Once you’ve set up SSH keys, you can log into your servers without needing to enter a password every time. If you add a passphrase to your private key for extra security, you can use an ssh-agent utility. The agent loads your decrypted key into memory, allowing you to use it for multiple connections during a single session after entering your passphrase just once.

• Automation: SSH keys are essential for automated processes. Tools used in CI/CD pipelines, configuration management (like Ansible), and backup scripts need to connect to remote servers without human intervention. Storing passwords in scripts is a major security risk. SSH keys provide a secure and non-interactive way for these automated systems to authenticate.

Types of SSH Keys

Several different cryptographic algorithms can be used to generate SSH keys. The most common ones are RSA, ECDSA, and Ed25519.

For most use cases, Ed25519 is the recommended SSH key algorithm. It offers the best combination of security and performance. It is more secure than ECDSA and much faster than RSA. Most modern operating systems and SSH clients have supported Ed25519 for several years.

You should only choose a different algorithm if you need to connect to a very old system that doesn’t support Ed25519. In such cases, a 4096-bit RSA key is a secure and widely compatible alternative.

Before you begin, ensure you have the following prerequisites in place. To follow this tutorial, you will need:

• An Ubuntu System: You should have access to a server or a local machine running a recent Long-Term Support (LTS) version of Ubuntu, such as 20.04, 22.04, or 24.04. The instructions provided are tailored for these versions and may vary on other systems.
• A Non-Root User with sudo Privileges: All commands that require administrative privileges will be preceded by sudo. This practice enhances security by preventing accidental system-wide changes. You must have an account that is configured to execute commands with these elevated permissions.
• OpenSSH Client and Server: Secure remote access is managed through SSH. You will need openssh-client to initiate connections and openssh-server to receive them. Most Ubuntu Desktop and Server installations include these packages by default.

Verifying and Installing OpenSSH

First, check the status of the ssh service, which is managed by openssh-server.

sudo systemctl status ssh

If the service is active, the output will indicate that it is “active (running)”. If it is not installed, you will see a “Unit ssh.service could not be found” message.

If they are not installed, run the following commands to install openssh-server and openssh-client:

sudo apt update
sudo apt install openssh-server openssh-client

Once the installation is complete, the SSH service will start automatically. With these prerequisites in place, you are now ready to proceed with the main steps of the tutorial.

The first step in setting up SSH key authentication is to generate a key pair on your local client machine. This pair consists of a private key, which you must keep secret, and a public key, which you can share.

The ssh-keygen Command

To generate a key pair, use the ssh-keygen command. While older systems defaulted to RSA keys, the modern recommended standard is Ed25519 due to its superior performance and security.

Open a terminal on your local machine and run the following command:

ssh-keygen -t ed25519

This command initiates the key generation process with the -t flag specifying the key type. You will see the first prompt:

Output
Generating public/private ed25519 key pair.
Enter file in which to save the key (/home/your_home/.ssh/id_ed25519):

You can press ENTER to accept the default file path and name. The keys will be stored in the hidden .ssh directory within your home folder. If you need to manage multiple keys, you can specify an alternate path here.

If a key pair already exists at the specified location, you will be asked if you want to overwrite it. Overwriting a key is a destructive action and will prevent you from authenticating with the old key, so proceed with caution.

Using Passphrases

Next, you will be prompted to enter a passphrase:

Output
Enter passphrase (empty for no passphrase):

A passphrase adds an extra layer of security to your private key. If your private key is ever compromised, an attacker would still need the passphrase to use it.

Here’s a breakdown of the pros and cons:

Recommendation: For interactive use, always use a strong passphrase. The inconvenience can be easily managed with an SSH agent, which we will cover in a later section. For automated systems where no user is present to enter a passphrase, you may need to omit it, but you must strictly limit the key’s permissions on the server.

After confirming your passphrase, the process will complete, and you will see output confirming the key generation:

Output
Your identification has been saved in /home/your_home/.ssh/id_ed25519
Your public key has been saved in /home/your_home/.ssh/id_ed25519.pub
The key fingerprint is:
SHA256:O3E2/c3nHCq+i4vIT0c+E+/bcfIs43hLi2dERbYtDb0 user@host
The key's randomart image is:
+--[ED25519 256]--+
|      .o*B+o     |
|     . =o+==.    |
|      ..o. o.    |
|       E. . o    |
|        S  . .   |
|         .  .    |
|        . o..    |
|       . ooo..   |
|        .=+o.    |
+----[SHA256]-----+

Understanding Key Files

You now have two new files in your ~/.ssh directory:

• id_ed25519: This is your private key. Treat it like a password. Never share it with anyone or expose it in an unsecured location. It is the key that proves your identity.
• id_ed25519.pub: This is your public key. It is designed to be shared and can be safely distributed. You will place this key on any server you want to access. The server uses the public key to verify the signature provided by your private key.

With your key pair generated, you are ready to deploy the public key to your server.

To enable key-based authentication, you must copy your public key to the target server and add it to the user account you want to log into. The server will then associate your public key with that account.

The Easy Way: Using ssh-copy-id

The ssh-copy-id command is the simplest and most recommended method for deploying your public key. It is available on most Linux, macOS, and Windows (via WSL) systems. This method requires you to have temporary password-based SSH access to the server.

The syntax is:

ssh-copy-id username@remote_host

Replace username with your user on the remote server and remote_host with its IP address or domain name.

The first time you connect, you may see a host authenticity warning. Type yes and press ENTER. The utility will then prompt you for the user’s password:

Output
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
username@203.0.113.1's password:

Enter the password. The ssh-copy-id tool will connect to the server, create the ~/.ssh directory and authorized_keys file if they don’t exist, and append your public key to the file with the correct permissions.

Output
Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'username@203.0.113.1'"
and check to make sure that only the key(s) you wanted were added.

Your key is now deployed.

The Manual Method

If ssh-copy-id is not available, or if password-based authentication is already disabled, you must add the key manually.

1. Get your public key string. Display the public key on your local machine:

   cat ~/.ssh/id_ed25519.pub
   
   

   Copy the entire output, which starts with ssh-ed25519 AAAA...

2. Log into your remote server. Use whatever method is available (e.g., a web console).

3. Append the key to authorized_keys. On the server, run the following command. This will create the necessary directory and file if they don’t exist and set the correct permissions.

   mkdir -p ~/.ssh && chmod 700 ~/.ssh && touch ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys
   
   

   Now, append your public key. Use the echo command and paste the key string you copied.

   echo "public_key_string" >> ~/.ssh/authorized_keys
   
   

   Ensure you use >> (append) and not > (overwrite) to avoid deleting existing keys.

Testing the Connection

After deploying your key, verify that key-based authentication is working. From your local machine, run:

ssh username@remote_host

• If you set a passphrase for your key, you will be prompted to enter it now.
• If you did not set a passphrase, you should be logged in immediately without a password prompt.

If you are logged in successfully, your setup is complete. You can now proceed to harden your server’s security.

As you work with more servers and services (like GitHub or GitLab), you may need to manage multiple SSH key pairs. An SSH client configuration file and an SSH agent can simplify this process significantly.

Client Configuration

You can configure your SSH client to use specific keys for different hosts by creating and editing the ~/.ssh/config file. This file allows you to create aliases for connections, specifying the hostname, user, and which private key to use.

Create the file if it doesn’t exist:

touch ~/.ssh/config

Open the file in a text editor and add entries for your hosts. For example:

# Work Server
Host work-server
    HostName 198.51.100.5
    User admin
    IdentityFile ~/.ssh/id_rsa_work

# Personal Project Server
Host personal-project
    HostName project.example.com
    User dev
    IdentityFile ~/.ssh/id_ed25519

# GitHub
Host github.com
    HostName github.com
    User git
    IdentityFile ~/.ssh/id_ed25519_github

With this configuration:

• ssh work-server will automatically connect to 198.51.100.5 as the admin user, using the id_rsa_work key.
• ssh personal-project connects to project.example.com as dev with the id_ed25519 key.

This eliminates the need to remember IP addresses, usernames, or specify the key with the -i flag for every connection.

SSH Agent

If you secured your private keys with passphrases, an SSH agent is essential for convenience. The agent is a background program that securely holds your decrypted private keys in memory, so you only need to enter your passphrase once per session.

1. Start the agent. To start the agent for your current terminal session, run:

   eval "$(ssh-agent -s)"
   
   

   You should see a confirmation with the agent’s process ID.

2. Add your key to the agent. Use the ssh-add command to add your private key:

   ssh-add ~/.ssh/id_ed25519
   
   

   You will be prompted to enter the passphrase for this key. Once entered, the agent will hold the decrypted key.

Now, any subsequent SSH connection that uses this key during your session will authenticate automatically without asking for the passphrase again. The agent will automatically provide the key for authentication.

Once you have confirmed that key-based authentication is working correctly, you should harden your server’s SSH configuration to enhance security. The two most important steps are disabling password authentication and disabling direct root login.

These changes are made in the SSH daemon configuration file, /etc/ssh/sshd_config. Open it with sudo privileges:

sudo nano /etc/ssh/sshd_config

Disabling Password Authentication

Disabling password authentication protects your server from brute-force attacks, where attackers try to guess user passwords.

Find the PasswordAuthentication directive. If it is commented out (starts with a #), uncomment it by removing the #. Set its value to no:

...
PasswordAuthentication no
...

This change ensures that only clients with a valid SSH key can authenticate.

Disabling Root Login

Allowing direct root login over SSH is a security risk because the root user is a known, high-value target. It’s a best practice to log in as a regular user and elevate privileges with sudo when needed.

Find the PermitRootLogin directive and set its value to no:

...
PermitRootLogin no
...

This prevents anyone from logging in directly as the root user over SSH.

Applying Changes

After making your changes, save the file and exit the editor (press CTRL+X, then Y, then ENTER in nano).

To apply the new configuration, you must restart the SSH service:

sudo systemctl restart ssh

To verify the changes without getting locked out, open a new terminal window and try to connect to the server. Your key-based login should work. You can also try connecting from a machine that does not have your SSH key, or try ssh root@remote_host, to confirm that password and root logins are correctly denied. Once you have verified the new settings, you can safely close your original session.

Beyond the standard setup, SSH offers powerful features for further hardening your security posture. These advanced techniques provide granular control over key usage, leverage hardware-based authentication, and enable secure access to remote network services.

Hardware Keys

For the highest level of security, you can use a physical hardware key, such as a YubiKey or other FIDO2/U2F-compliant device. This method stores your private key on a tamper-resistant hardware device, meaning the key material itself never touches your computer’s disk. Authentication requires not just possession of the key but also a physical action (touching the device), providing robust protection against key theft and malware.

1. Generate the Key: Ensure your hardware key is plugged into your machine. Use the ssh-keygen command with the type ed25519-sk or ecdsa-sk:

   ssh-keygen -t ed25519-sk
   
   

   This will prompt you to touch your hardware key to confirm the operation. The process generates two files: the public key (id_ed25519_sk.pub) and a private key handle (id_ed25519_sk). The private key handle is not the key itself but a pointer to the key stored securely on the hardware device.

2. Deploy the Public Key: Copy the contents of id_ed25519_sk.pub to your server’s ~/.ssh/authorized_keys file, just as you would with a standard key.

3. Authenticate: When you initiate an SSH connection using this key, your SSH client will prompt you to touch your hardware key to authorize the session. This physical interaction ensures that an attacker cannot use your key without physical access to your device.

Restricting Key Access

You can apply fine-grained restrictions to individual keys by adding options at the beginning of their corresponding lines in the ~/.ssh/authorized_keys file on the server. This is especially useful for automating tasks or granting limited access.

The format is: options public-key-string

Here are some of the most practical options:

Practical Example:

Imagine you need a key for an automated backup script running on a server at 198.51.100.5. You want this key to only be able to run the backup script and nothing else. Your entry in authorized_keys would look like this:

from="198.51.100.5",command="/opt/scripts/backup.sh",no-port-forwarding,no-agent-forwarding,no-X11-forwarding ssh-ed25519 AAAAC3... backup-script-key

This configuration ensures the key can only be used from the specified IP to run a single, predefined command, with all other SSH features disabled.

SSH Agent Forwarding

SSH agent forwarding is a mechanism that allows you to use your local SSH keys to authenticate to a second server from a machine you are already connected to, without copying your private key to the intermediate server.

• Use Case: The most common scenario is connecting through a “bastion” or “jump” host. You SSH from your local machine (A) to a bastion server (B), and from there, you need to connect to an internal server ©. Agent forwarding lets you use the key from machine A to authenticate to machine C.

• How to Use: To enable agent forwarding for a single connection, use the -A flag:

  ssh -A user@bastion-host
  
  

  Once logged into bastion-host, you can then SSH to the internal server, and your local agent will handle the authentication:

  ssh user@internal-server
  
  

• Security Implications: While convenient, agent forwarding carries a significant security risk. If the intermediate server (bastion-host) is compromised, an attacker with root privileges on that machine can hijack your agent’s connection socket. This would allow them to use your forwarded credentials to log in to any other server that your key has access to, for as long as your session is active. Only use agent forwarding when connecting to servers you fully trust. A more secure modern alternative is to use the ProxyJump directive in your ~/.ssh/config file.

SSH Tunneling (Port Forwarding)

SSH tunneling, or port forwarding, creates an encrypted channel to securely route network traffic between your local machine and a remote server. This is useful for accessing services that are not exposed to the public internet or for encrypting traffic for legacy applications.

Local Port Forwarding

Local port forwarding (-L) allows you to access a service on a remote network as if it were running on your local machine.

• Use Case: Accessing a database on a remote server that only allows connections from localhost (itself).

• Syntax: ssh -L local_port:destination_host:destination_port user@ssh_server

• Example: To forward port 3306 (MySQL) on remote_server to your local port 8888, run:

  ssh -L 8888:localhost:3306 user@remote_server
  
  

  Now, you can connect your local database client to localhost:8888, and the traffic will be securely tunneled to port 3306 on remote_server.

Remote Port Forwarding

Remote port forwarding (-R) does the opposite: it exposes a service running on your local machine to the remote server’s network.

• Use Case: Temporarily showing a web application running on your local development machine (localhost:3000) to a colleague who can access the remote server.

• Syntax: ssh -R remote_port:destination_host:destination_port user@ssh_server

• Example: To expose your local web server to port 9999 on remote_server, run:

  ssh -R 9999:localhost:3000 user@remote_server
  
  

Your colleague can now access http://remote_server:9999 in their browser, and the requests will be forwarded to your local machine on port 3000. Be cautious with this feature, as it can expose local services to a remote network.

Setting up SSH keys is a critical first step, but maintaining a secure configuration over time is just as important. Adhering to best practices for permissions, key rotation, and backups will ensure the long-term integrity of your server access.

File Permissions

Both the SSH client and the server daemon (sshd) enforce strict permission checks on key files to prevent unauthorized users from accessing them. Incorrect permissions are one of the most common reasons for authentication failures.

You must set the following permissions on both the client and server where applicable:

To apply these permissions, use the chmod command:

# On the server and client
chmod 700 ~/.ssh

# On the server
chmod 600 ~/.ssh/authorized_keys

# On the client (for your private key)
chmod 600 ~/.ssh/id_ed25519

Key Rotation

Key rotation is the process of periodically retiring old SSH keys and replacing them with new ones. This is a critical security practice that limits the damage if a private key is ever compromised. A stolen key provides indefinite access until it is removed from the server.

Rotation Policy: The frequency of rotation depends on your organization’s security standards. A common policy is to rotate keys annually. For highly sensitive infrastructure, rotation might occur quarterly or even more frequently.

Rotation Process:

1. Generate a new key pair on your local machine (ssh-keygen -t ed25519).
2. Deploy the new public key to the ~/.ssh/authorized_keys file on all relevant servers using ssh-copy-id.
3. Test the connection thoroughly using the new key to ensure it works correctly.
4. Remove the old public key from the ~/.ssh/authorized_keys file on each server.
5. Securely delete the old private key and public key files from your local machine.

Backup and Recovery

Losing your private key means losing access to every server configured to use it. A secure backup strategy is essential.

• Secure Storage: Never store backups of your private key in an unencrypted format. Use a trusted password manager (like Bitwarden or 1Password), an encrypted USB drive, or a secure cloud storage service that provides client-side encryption.
• Recovery Plan: In addition to backups, have a secondary method for accessing your server. This is often called a “break-glass” procedure. The most common method is using the web-based console provided by your cloud or hosting provider. This console access allows you to log in (often with a password) and manually add a new public key to your authorized_keys file if you lose your original.

Even with a careful setup, you may encounter issues. Most problems fall into a few common categories related to permissions, network connectivity, or key mismatches.

“Permission denied” Errors

This is the most frequent error and almost always points to a problem with the key or file permissions. You will see a message like this:

Output
username@remote_host: Permission denied (publickey).

Common Causes and Solutions:

1. Incorrect Server-Side Permissions: The SSH daemon on the server will reject the connection if the ~/.ssh directory or ~/.ssh/authorized_keys file has permissions that are too open. Run these commands on the server for the user you are connecting as:

   chmod 700 ~/.ssh
   chmod 600 ~/.ssh/authorized_keys
   
   

2. Incorrect Client-Side Permissions: Your SSH client will not use a private key file that other users on your system can read. Ensure your private key is protected:

   chmod 600 ~/.ssh/id_ed25519
   
   

3. Key Mismatch: The public key on the server may not correspond to the private key your client is offering. Verify that the entire contents of your local .pub file are present on a single line in the server’s authorized_keys file.

Connection Problems

If you see errors like Connection refused or the connection times out, the problem is likely network-related or an issue with the SSH service itself.

Common Causes and Solutions:

1. Firewall is Blocking the Port: A firewall on the server is the most common culprit. On Ubuntu, you can check the status of UFW (Uncomplicated Firewall) and ensure the SSH port (22) is allowed.

   # Check firewall status on the server
   sudo ufw status
   
   # If SSH is not allowed, add the rule
   sudo ufw allow ssh
   
   

2. SSH Daemon is Not Running: The SSH service (sshd) may be inactive on the server. Check its status:

   # Check the status of the ssh service on the server
   sudo systemctl status ssh
   
   

   If it is not active, you can try starting or restarting it:

   sudo systemctl restart ssh
   
   

3. Incorrect IP Address or Port: Double-check that you are using the correct server IP address. If the server’s SSH daemon is listening on a non-standard port (e.g., 2222), you must specify it with the -p flag:

   ssh -p 2222 username@remote_host
   
   

Using Debug Mode

The most powerful tool for diagnosing any SSH connection issue is the verbose output mode. Using the -v flag tells the SSH client to print detailed debug information about the connection process.

ssh -v username@remote_host

You can increase the level of detail with -vv or -vvv. The output will show:

• Which configuration files are being read.
• The steps of the cryptographic handshake.
• Which private keys the client is attempting to use (Offering public key...).
• The server’s response to each key.

This detailed log will almost always reveal the exact point of failure, whether it’s not finding the right key, failing on permissions, or being rejected by the server.

SSH is often seen as a simple command-line tool for logging into a remote server. For AI and machine learning professionals, however, it’s a foundational technology that unlocks secure, efficient, and automated workflows. Whether you’re running a Jupyter Notebook on a powerful GPU-enabled server, automating a complex training pipeline, or empowering an AI agent to manage its own infrastructure, SSH is the secure backbone that makes it all possible.

Tunneling for AI Notebooks (Jupyter, VS Code, etc.)

AI development often requires more computational power than a local laptop can provide, especially when working with large datasets and complex models. This means running development environments like Jupyter Notebooks or VS Code on powerful remote servers. The challenge is accessing these web-based or client-server tools securely over the internet.

This is where SSH tunneling, also known as port forwarding, becomes essential. An SSH tunnel creates a secure, encrypted connection between a port on your local machine and a port on the remote server. Any traffic you send to the local port is automatically and securely forwarded through the SSH connection to the specified port on the remote machine. This allows you to interact with a remote application as if it were running locally.

Practical Example: Securely Accessing a Remote Jupyter Notebook

Imagine you’ve launched a Jupyter Notebook server on a remote machine. By default, Jupyter protects your session with an authentication token and runs on port 8888. While the token adds a layer of security, exposing the service directly to the internet is still a risk. Using an SSH tunnel is the standard practice for creating a secure connection.

You can use a local SSH port forward to create this secure tunnel, mapping port 8888 on your local machine to port 8888 on the remote server.

1. Establish the Tunnel: Open a terminal on your local machine and run the following command:

   ssh -L 8888:localhost:8888 user@remote-server-ip
   

   Let’s break down the -L flag’s arguments:

   • 8888: The port you will access on your local machine.
   • localhost: The destination host the remote server should connect to. Since Jupyter is running on the remote server itself, localhost is the correct value.
   • 8888: The destination port on the remote server where the Jupyter Notebook is listening.
   • user@remote-server-ip: Your standard SSH login credentials for the remote server.

2. Access Jupyter: Once the SSH connection is active, open your local web browser and navigate to http://localhost:8888. The SSH tunnel will securely forward your request to the Jupyter server. You will still need to enter the Jupyter authentication token the first time you connect.

Integrating with VS Code Remote Development

The same principle applies to modern developer tools like Visual Studio Code. The Remote - SSH extension automates the secure connection for editing files, using a terminal, and debugging code directly on the remote machine.

However, it’s important to note that the extension handles the core IDE experience but doesn’t automatically forward every web-based application you might run on the server (like a separate Jupyter server or a Streamlit dashboard). For these cases, you can use VS Code’s integrated Port Forwarding feature. This allows you to manually or automatically forward any additional required ports through the existing secure SSH connection, providing the same benefit as the -L flag in a command-line client.

Automating AI Pipelines with SSH

Beyond interactive sessions, SSH is a powerful tool for automating multi-step AI workflows. Reproducibility and efficiency are key in machine learning, and scripting your pipeline—from data transfer to model training and results retrieval—is a cornerstone of modern MLOps.

An automated AI pipeline typically involves three main stages, all of which can be scripted using SSH commands.

1. Upload Code and Data: Before you can run a job, you need to get your scripts and datasets onto the remote server. While scp (secure copy) is suitable for single files, rsync (remote sync) is generally preferred for its efficiency in syncing directories. For large-scale AI workflows, specialized tools are often used.

   • Using rsync to sync a directory with progress:

     rsync -avz --progress ./my-project/ user@remote-server-ip:/home/user/project/
     

   • Specialized Tools: For very large datasets, consider tools like Rclone, which is optimized for syncing files with cloud storage providers (S3, Google Cloud Storage, etc.), or DVC (Data Version Control), which integrates with Git to manage large data files and models.

2. Trigger Training or Inference Jobs: You can execute a command on a remote server without opening an interactive shell by simply appending it to your SSH command. This is perfect for launching a training script.

   • Execute a Python script remotely:

     ssh user@remote-server-ip "python /home/user/project/train.py --epochs 50 --batch-size 32"
     

   The command will execute, and its standard output will be printed to your local terminal.

3. Download Results: After the job is complete, you can use scp or rsync again to download the resulting artifacts, such as trained model weights, logs, or performance metrics.

   • Download a trained model file:

     scp user@remote-server-ip:/home/user/project/models/final_model.pth ./local-results/
     

Tools for Advanced Automation

While simple shell scripts work well, you can achieve more sophisticated automation using dedicated tools and libraries.

• Fabric: A high-level Python library designed for application deployment and systems administration tasks over SSH.
• Paramiko: A lower-level Python library that implements the SSHv2 protocol, providing granular control for building custom SSH applications.
• CI/CD Workflows: These scripted SSH commands can be integrated directly into CI/CD platforms like GitHub Actions or Jenkins to build fully automated MLOps pipelines.

Using SSH with AI Agents

One of the most forward-looking applications of SSH is empowering AI agents to interact with and manage remote infrastructure. AI agents, such as those built with frameworks like LangChain or AutoGPT, are autonomous systems that can reason, create plans, and execute actions to achieve a goal.

By giving an agent access to SSH as a “tool,” you enable it to perform complex infrastructure management tasks. However, it’s important to frame this correctly: although direct SSH access is possible for experimentation, in production environments, it is far more common for agents to interact with more constrained, higher-level infrastructure APIs (like the Kubernetes API, cloud provider SDKs, or job schedulers like Slurm) rather than being granted raw SSH access. This approach provides a more secure and auditable control plane.

Example Use Cases for SSH-Enabled AI Agents

• Dynamic Resource Management: An agent could be tasked with running a training job efficiently. It could perform the following steps autonomously:

  • SSH into a list of available servers in a cluster.
  • Run the nvidia-smi command on each to check GPU availability, utilization, and memory.
  • Analyze the results to select the optimal server for the job.
  • Use scp to transfer the necessary code and data to the selected server.
  • Trigger the training script via an SSH command.

• Automated Model Deployment: An agent could handle the entire deployment process. Given the instruction “deploy the latest model to production,” it could:

  • SSH into the production inference server.
  • Pull the latest application code from a Git repository.
  • Download the newest model weights from a model registry.
  • Restart the inference service (e.g., systemctl restart my-inference-api) to load the new model.

Crucial Security Considerations

Granting an AI agent SSH access carries significant security risks. It is imperative to follow the principle of least privilege. Under no circumstances should an AI agent ever be given root access to a system.

By implementing these safeguards, you can experiment with the power of AI agents to manage infrastructure while maintaining a secure and controlled environment.

1. How do I generate SSH keys on Ubuntu 22.04?

You generate an SSH key pair using the ssh-keygen command-line tool. The recommended modern algorithm is Ed25519 due to its superior security and performance.

Open your terminal and run the following command:

ssh-keygen -t ed25519 -C "your_email@example.com"

• -t ed25519: This flag specifies the type of key to create.
• -C "your_email@example.com": This adds a comment to the key, typically an email, to help you identify it later.

After executing the command, you’ll be prompted for two things:

1. File location: Press Enter to accept the default location, which will be inside the ~/.ssh/ directory (e.g., /home/user/.ssh/id_ed25519).
2. Passphrase: You will be asked to enter and confirm a passphrase. This is a crucial security step. A passphrase encrypts your private key on your disk, so even if someone gains access to the file, they cannot use it without the passphrase.

Once complete, ssh-keygen will confirm that your public and private keys have been saved.

2. Where are SSH keys stored in Ubuntu?

By default, the SSH client stores its key pairs and configuration in the .ssh directory within your user’s home directory (~/.ssh/). This directory is hidden, as indicated by the leading dot.

Inside this directory, you will find two primary files for each key pair:

1. Private Key (id_ed25519): This is your secret key. It must be protected and should never be shared with anyone or exposed publicly. The private key is used to prove your identity to a remote server.
2. Public Key (id_ed25519.pub): This key is designed to be shared. You copy the contents of this file to remote servers to grant yourself access. The server uses this key to verify a connection request signed by your corresponding private key.

3. How do I copy my SSH key to a remote server?

The easiest and most recommended method is to use the ssh-copy-id command. This utility automatically copies your public key to the remote server, appends it to the correct ~/.ssh/authorized_keys file, and sets the proper directory and file permissions.

To use it, run the command below, substituting your remote username and the server’s IP address or hostname.

ssh-copy-id username@remote_host

You will be asked for the user’s password on the remote server one final time. After successful authentication, your key will be copied, and subsequent SSH logins will be passwordless.

If ssh-copy-id is not available, you can perform the steps manually by copying your public key and appending it to the ~/.ssh/authorized_keys file on the remote server.

4. What key type should I use for SSH in 2025 (RSA vs Ed25519)?

For all new keys, you should use Ed25519. It is a modern elliptic curve algorithm that offers better security and significantly better performance than the older RSA standard.

While RSA is not considered broken, Ed25519 is the superior choice for nearly all use cases.

Use Ed25519 unless you have a specific requirement to connect to an outdated system that does not support it.

5. How do I disable password login in Ubuntu SSH?

Disabling password-based authentication is a critical security measure that protects your server from brute-force attacks. Once you have confirmed that your key-based login works, you should disable passwords on your remote server.

1. SSH into your remote server.

2. Open the SSH daemon’s configuration file using a text editor with root privileges:

   sudo nano /etc/ssh/sshd_config
   

3. Inside the file, locate the PasswordAuthentication directive. Uncomment it (remove the #) if necessary and set its value to no.

   PasswordAuthentication no
   

4. Ensure that PubkeyAuthentication is set to yes to allow key-based logins.

   PubkeyAuthentication yes
   

5. Save the file (Ctrl+O, Enter) and exit the editor (Ctrl+X).

6. Restart the SSH service for the changes to take effect:

   sudo systemctl restart sshd
   

6. How do I fix “permissions are too open” SSH error?

This common error occurs because the SSH client enforces strict security policies. Your private key and its parent directory must not be writable or accessible by other users on your system. If they are, SSH will refuse to use the key to prevent potential security breaches.

The error message typically looks like this: Permissions 0644 for '/home/user/.ssh/id_ed25519' are too open.

To fix this, you must set the correct file permissions using the chmod command.

• On your local machine, run these commands:

  chmod 700 ~/.ssh
  chmod 600 ~/.ssh/id_ed25519
  

  • 700 gives read, write, and execute permissions only to you, the owner.
  • 600 gives read and write permissions only to you, the owner.

• On your remote server, ensure the permissions are also correct:

  chmod 700 ~/.ssh
  chmod 600 ~/.ssh/authorized_keys
  

7. Can I use the same SSH key for multiple servers?

Yes, it is common practice to use the same SSH key pair to access multiple servers. The process involves generating a single key pair on your local workstation and then copying the public key (id_ed25519.pub) to every server you need to manage.

This approach is convenient because you only need to manage one private key. However, it comes with a security trade-off:

• Convenience: Manage one identity across many systems.
• Risk: If your single private key is ever stolen or compromised, an attacker gains access to all servers that trust its public key.

For enhanced security, especially in production or business environments, it’s often better to use separate key pairs for different roles or environments (e.g., a personal key, a work key for development, and another for production). This practice, known as privilege separation, limits the potential damage if one key is compromised.

You have now successfully configured SSH key-based authentication on your Ubuntu 22.04 server. By generating a secure key pair and disabling password-based logins, you have significantly improved your server’s security against brute-force attacks and enabled a faster, more convenient workflow for remote administration. This key-based access is a foundational skill for managing systems and automating tasks securely.

To build on what you’ve learned, consider exploring advanced SSH features like using a configuration file (~/.ssh/config) to manage multiple connections or an ssh-agent to handle your passphrase. For a deeper dive into SSH capabilities, review our comprehensive guides:

• SSH Essentials Guide.
• How to Use SSH to Connect to a Remote Server (Step-by-Step Guide)
• How to Create an SSH Key in Linux: Easy Step-by-Step Guide