Secure Your Linux Server: Harden SSH Access with Public Key Authentication

When you manage a Linux server, SSH (Secure Shell) is the gateway you use every day. Unfortunately, the default password‑based login is a favorite target for brute‑force attacks. Switching to public key authentication not only stops password‑guessing bots in their tracks, it also gives you finer control over who can log in and from where. In this guide we’ll walk you through the entire hardening process—from generating a key pair on your workstation to locking down the SSH daemon so that passwords are no longer accepted. By the end, you’ll have a server that only trusted keys can access, and you’ll understand the common pitfalls to avoid.

What You’ll Need

• A Linux server (any distribution) with a user that has sudo or root privileges.
• A workstation (Linux, macOS, or Windows with WSL/PowerShell) where you can generate an SSH key pair.
• Basic familiarity with the terminal and a text editor (nano, vim, or your favorite IDE).
• Internet access to fetch updates and optional security tools like fail2ban.

Step 1: Generate a Strong SSH Key Pair

Open a terminal on your local machine and run:

ssh-keygen -t ed25519 -a 100 -C "your_email@example.com"

The -t ed25519 flag creates a modern, fast, and secure key type. The -a 100 option adds 100 rounds of key‑derivation (KDF) to protect the private key with a passphrase. When prompted, accept the default file location (~/.ssh/id_ed25519) and choose a strong passphrase. You now have two files:

• id_ed25519 – your private key (keep it secret!)
• id_ed25519.pub – the public key you will copy to the server.

Step 2: Copy the Public Key to the Server

The simplest way is to use ssh-copy-id:

ssh-copy-id -i ~/.ssh/id_ed25519.pub user@your.server.com

If ssh-copy-id is not available, you can manually append the key:

cat ~/.ssh/id_ed25519.pub | ssh user@your.server.com "mkdir -p ~/.ssh && chmod 700 ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"

Make sure the ~/.ssh directory and authorized_keys file have the correct permissions (700 and 600 respectively). This step ensures the server trusts the public key you just generated.

Step 3: Harden the SSH Daemon Configuration

Edit the SSH daemon’s main config file:

sudo nano /etc/ssh/sshd_config

Locate and adjust the following directives (add them if they are missing):

# Use only modern protocols
Protocol 2

# Restrict authentication methods
PasswordAuthentication no
ChallengeResponseAuthentication no
PubkeyAuthentication yes

# Disallow direct root login
PermitRootLogin no

# Limit which users may log in (optional but recommended)
AllowUsers youruser

# Optional: enforce key expiration or certificate use
# ExpireAfter 30d (requires OpenSSH 8.8+)

Save the file and exit. These settings tell SSH to accept only public‑key logins, reject passwords, and prevent the root account from logging in directly.

Step 4: Restart the SSH Service

Apply the new configuration by restarting the daemon:

sudo systemctl restart sshd   # Debian/Ubuntu
# or
sudo systemctl restart sshd.service   # RHEL/CentOS/Fedora

Immediately test a new connection in a separate terminal window before you close your existing session. This prevents being locked out if something went wrong.

Step 5: Verify Password Authentication Is Disabled

From your workstation, try to log in without specifying a key:

ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no user@your.server.com

If the server correctly denies the login, you’ll see a message like “Permission denied (publickey).” If you can still log in with a password, double‑check the sshd_config file for duplicate entries that might be overriding your changes.

Step 6: Tighten File Permissions on the Server

Even with a hardened sshd_config, insecure permissions can expose your keys. Run the following commands on the server:

sudo chmod 700 /home/youruser/.ssh
sudo chmod 600 /home/youruser/.ssh/authorized_keys
sudo chown -R youruser:youruser /home/youruser/.ssh

If you created a dedicated ssh group for key‑based users, make sure the group ownership matches.

Step 7: Add Extra Layers of Defense (Optional but Recommended)

Public key authentication is strong, but adding a second barrier makes attacks even harder.

• Fail2Ban: Install and enable a jail for SSH to ban IPs after repeated failed attempts.

  sudo apt-get install fail2ban   # Debian/Ubuntu
  sudo systemctl enable --now fail2ban

• Firewall Rules: Limit SSH to known IP ranges using ufw or iptables.

  # Example with ufw – allow only from 203.0.113.0/24
  sudo ufw allow from 203.0.113.0/24 to any port 22
  sudo ufw enable

• Two‑Factor Authentication (2FA): For ultra‑sensitive environments, consider adding google-authenticator as a second factor.

Common Mistakes to Avoid

1 Leaving PasswordAuthentication yes in the config. A stray line can re‑enable password logins.
2 Incorrect file permissions. If .ssh is world‑readable, SSH will ignore the authorized_keys file.
3 Copying the private key instead of the public key. Never place id_ed25519 on the server.
4 Editing the wrong sshd_config file. Some distributions have /etc/ssh/sshd_config.d/ snippets that override settings.
5 Testing only from the same session. Always open a new terminal to confirm you can still connect before closing the original.

Tips and Tricks

Use a dedicated SSH key per server. It limits the blast radius if a key is ever compromised.
Enable SSH agent forwarding cautiously. It’s convenient for hopping between servers, but only enable it on trusted jump hosts.
Rotate keys regularly. Generate a new pair every 6‑12 months and update authorized_keys accordingly.
Audit your authorized_keys file. Periodically remove stale entries and add comments (e.g., # alice@laptop 2024-03-01) to keep track of ownership.

Frequently Asked Questions

Can I still use password login for a specific user?

Yes. Create a separate Match User block at the end of sshd_config and set PasswordAuthentication yes only for that user. However, this re‑introduces a weak entry point, so it’s best avoided.

What if I lose my private key?

If you lose the private key and have no backup, you’ll be locked out. The recovery path is to log in via another authorized key (if you have one) or use console access (e.g., through a cloud provider’s web console) to add a new public key to ~/.ssh/authorized_keys.

Do I need to restart SSH after every change?

Any change to sshd_config requires a restart or reload. Use sudo systemctl reload sshd for a graceful reload, but a full restart is safer after major changes like disabling passwords.

Conclusion

Hardening SSH with public key authentication is one of the most effective, low‑cost ways to protect a Linux server from automated attacks. By generating a strong key pair, correctly placing the public key, tightening sshd_config, and reinforcing the setup with firewalls or fail2ban, you create a robust barrier that stops most malicious login attempts. Remember to double‑check permissions, test from a fresh session, and keep your keys under regular review. With these steps in place, you can breathe easier knowing that only the right keys—and you—can walk through the SSH door.

Photo by FlyD on Unsplash