Introduction to SSH Key Management

What SSH Keys Are

SSH (Secure Shell) keys are cryptographic credentials used for authenticating users and machines in secure network communications. Unlike passwords, SSH keys provide a more secure and scalable method for accessing remote systems.

The Basics of SSH Authentication

Traditional Password Authentication:

ssh user@server.example.com
Password: ********

Problems with passwords:

Vulnerable to brute force attacks
Difficult to manage at scale
Often reused across systems
Can be intercepted or phished
No audit trail of usage

SSH Key Authentication:

ssh -i ~/.ssh/id_rsa user@server.example.com
# No password prompt - authenticated via key pair

Benefits of SSH keys:

Cryptographically strong (2048-4096 bit)
Immune to brute force attacks
Unique per user/system
Can be automated safely
Supports detailed logging

How SSH Keys Work

SSH uses asymmetric cryptography with a key pair:

Private Key:

Kept secret by the owner
Never transmitted over the network
Used to prove identity
Typically stored in ~/.ssh/id_rsa or ~/.ssh/id_ed25519
Should be protected with a passphrase

Public Key:

Shared freely
Stored on servers in ~/.ssh/authorized_keys
Used to verify identity
Safe to distribute widely

Authentication Process:

Client initiates connection to server
Server sends a challenge encrypted with the public key
Client decrypts challenge with private key
Client sends response proving key possession
Server grants access if response is correct

SSH Key Types

Modern SSH supports several key algorithms:

RSA (Rivest-Shamir-Adleman):

ssh-keygen -t rsa -b 4096 -C "user@example.com"

Most widely supported
Minimum 2048 bits (4096 recommended)
Slower than newer algorithms
Still secure when properly sized

Ed25519 (Edwards-curve Digital Signature Algorithm):

ssh-keygen -t ed25519 -C "user@example.com"

Modern, fast, and secure
Fixed 256-bit key size
Resistant to timing attacks
Recommended for new deployments
Not supported by very old systems

ECDSA (Elliptic Curve Digital Signature Algorithm):

ssh-keygen -t ecdsa -b 521 -C "user@example.com"

Good performance
Smaller key sizes than RSA
Some controversy over NIST curves
Less recommended than Ed25519

DSA (Digital Signature Algorithm):

Deprecated - do not use
Limited to 1024 bits
Vulnerable to weak random number generation
Disabled in OpenSSH 7.0+

Common Enterprise Security Problems

As organizations scale, SSH key management becomes increasingly complex and risky. Here are the most common challenges:

1. SSH Key Sprawl

The Problem:

Developers create keys for each project
Automated systems generate keys without tracking
Keys accumulate over years
No central inventory exists

Real-world scenario: A Fortune 500 company discovered over 3 million SSH keys across their infrastructure, with:

90% of keys untracked
10% of keys belonging to former employees
Thousands of keys with unknown ownership
No process for key rotation

Security risks:

Unauthorized access persists after employee departure
Compromised keys go undetected
Compliance violations (SOX, PCI DSS, HIPAA)
Impossible to audit access

2. Orphaned and Stale Keys

The Problem: Keys remain active long after they should be removed:

Employee leaves company → keys remain in authorized_keys
Contractor engagement ends → access persists
Service account keys never expire
Test keys left in production

Impact:

# Typical authorized_keys file
cat ~/.ssh/authorized_keys

ssh-rsa AAAAB3... john@laptop     # John left 2 years ago
ssh-rsa AAAAB3... deploy-bot      # Unknown origin
ssh-rsa AAAAB3... contractor      # Contract ended 6 months ago
ssh-rsa AAAAB3... backup-system   # System decommissioned
ssh-ed25519 AAAAC3... alice@work  # Current employee

Only 1 of 5 keys should still have access!

Consequences:

Former employees retain access
Increased attack surface
Compliance failures
Difficult incident response

3. Lack of Visibility and Auditing

The Problem: Organizations can’t answer basic questions:

Who has access to which servers?
When was a key last used?
Which keys are associated with which users?
What actions were performed with each key?

Audit challenges:

# SSH logs show key fingerprints, not identities
grep "Accepted publickey" /var/log/auth.log

Accepted publickey for root from 192.168.1.100 port 52234 ssh2: RSA SHA256:abc123...

Questions that can’t be answered:

Whose key is SHA256:abc123…?
Why did they access this server?
Is this access authorized?
What did they do after logging in?

4. Weak Key Management Practices

Common mistakes:

Shared keys:

# Team shares a single key - no individual accountability
ssh -i ~/.ssh/team_shared_key user@server

Keys without passphrases:

# Private key has no passphrase protection
ssh-keygen -t rsa -b 4096 -N ""

Keys in version control:

# Accidentally committed to Git
git log --all --full-history -- "*id_rsa*"

Hardcoded keys in scripts:

# Private key embedded in deployment script
echo "-----BEGIN RSA PRIVATE KEY-----" > /tmp/deploy_key

Long-lived or permanent keys:

# Key created in 2015, never rotated
ls -la ~/.ssh/id_rsa
-rw------- 1 user user 3243 Jan 15  2015 /home/user/.ssh/id_rsa

5. Privileged Access Challenges

The Problem: SSH keys often provide root or sudo access:

# Direct root access
ssh root@production-server

# Sudo access without password
user ALL=(ALL) NOPASSWD: ALL

Risks:

No approval workflow for privileged access
Permanent elevated privileges
Difficult to implement least privilege
No time-limited access
Compliance violations

6. Automation and Service Accounts

The Problem: Automated systems require SSH access:

CI/CD pipelines deploying code
Backup systems accessing servers
Monitoring tools collecting data
Configuration management (Ansible, Puppet)

Challenges:

# Service account key with broad access
# Used by: Jenkins, GitLab CI, backup system, monitoring
ssh-rsa AAAAB3... automation@company.com

Single key used by multiple systems
Keys stored in multiple locations
Difficult to rotate without breaking automation
No way to distinguish which system is connecting

SSH Certificate Authorities

SSH Certificate Authorities (CAs) solve many traditional SSH key management problems by introducing a trust model similar to TLS/SSL certificates.

How SSH CAs Work

Instead of distributing public keys to every server, SSH CAs issue short-lived certificates:

Traditional SSH:

User Public Key → Manually added to each server's authorized_keys

SSH CA:

User Public Key → Signed by CA → Certificate → Trusted by all servers

SSH CA Architecture

Components:

CA Key Pair:
- Private key signs certificates
- Public key trusted by all servers
- Stored securely (HSM recommended)
User Certificates:
- Short-lived (hours to days)
- Contain user identity and permissions
- Signed by CA private key
Host Certificates:
- Identify servers to clients
- Prevent man-in-the-middle attacks
- Eliminate “unknown host” warnings

Setting Up an SSH CA

Step 1: Generate CA Key Pair

# Create CA key pair
ssh-keygen -t ed25519 -f /secure/location/ssh_ca -C "SSH CA"

# Protect CA private key
chmod 400 /secure/location/ssh_ca

Step 2: Configure Servers to Trust CA

# Add CA public key to sshd_config
echo "TrustedUserCAKeys /etc/ssh/ca.pub" >> /etc/ssh/sshd_config

# Copy CA public key to server
scp /secure/location/ssh_ca.pub server:/etc/ssh/ca.pub

# Restart SSH daemon
systemctl restart sshd

Step 3: Issue User Certificates

# User generates key pair
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519

# User submits public key for signing
# (In practice, this goes through an approval workflow)

# CA signs the public key
ssh-keygen -s /secure/location/ssh_ca \
  -I "alice@example.com" \
  -n alice,admin \
  -V +8h \
  ~/.ssh/id_ed25519.pub

# Creates: ~/.ssh/id_ed25519-cert.pub

Step 4: User Connects with Certificate

# SSH automatically uses certificate if present
ssh alice@server.example.com

# Certificate is validated:
# - Signed by trusted CA?
# - Still valid (not expired)?
# - Principal matches username?
# - Extensions allow this action?

Certificate Contents

An SSH certificate contains:

# View certificate details
ssh-keygen -L -f ~/.ssh/id_ed25519-cert.pub

Output:

Type: ssh-ed25519-cert-v01@openssh.com user certificate
Public key: ED25519-CERT SHA256:abc123...
Signing CA: ED25519 SHA256:def456...
Key ID: "alice@example.com"
Serial: 0
Valid: from 2024-11-21T09:00:00 to 2024-11-21T17:00:00
Principals:
        alice
        admin
Critical Options: (none)
Extensions:
        permit-pty
        permit-user-rc
        permit-X11-forwarding
        permit-agent-forwarding
        permit-port-forwarding

Key fields:

Key ID: Human-readable identifier for logging
Valid: Time window for certificate validity
Principals: Usernames allowed for login
Extensions: Permissions granted

Benefits of SSH CAs

1. Centralized Trust:

One CA public key on all servers
No need to update authorized_keys files
Easy to add/remove servers

2. Short-Lived Credentials:

Certificates expire automatically
Typical validity: 8-24 hours
Reduces impact of key compromise
No need to manually revoke

3. Enhanced Auditing:

Key ID logged with every connection
Clear identity attribution
Detailed permission tracking

4. Flexible Permissions:

Different principals for different roles
Time-based access restrictions
Command restrictions via critical options
Source IP restrictions

5. Simplified Onboarding/Offboarding:

New employees: Issue certificate
Departing employees: Stop issuing certificates
No need to touch server configurations

Advanced SSH CA Features

Principal Mapping:

# Certificate allows multiple usernames
ssh-keygen -s ca_key -I "alice" -n alice,root,deploy -V +8h user_key.pub

# Alice can log in as any principal
ssh alice@server
ssh root@server
ssh deploy@server

Force Commands:

# Restrict certificate to specific command
ssh-keygen -s ca_key -I "backup" -n backup \
  -O force-command="/usr/local/bin/backup.sh" \
  -V +1h backup_key.pub

Source Address Restrictions:

# Certificate only valid from specific IPs
ssh-keygen -s ca_key -I "alice" -n alice \
  -O source-address=192.168.1.0/24 \
  -V +8h user_key.pub

Host Certificates:

# Sign server's host key
ssh-keygen -s ca_key -I "server.example.com" \
  -h -n server.example.com,192.168.1.10 \
  -V +365d /etc/ssh/ssh_host_ed25519_key.pub

# Clients trust CA for host verification
echo "@cert-authority *.example.com $(cat ca.pub)" >> ~/.ssh/known_hosts

Best Practices

Implementing robust SSH key management requires following industry best practices:

1. Key Rotation

Why rotate keys?

Limits exposure window if compromised
Removes orphaned access
Maintains security hygiene
Meets compliance requirements

Rotation schedule:

User keys: Every 90-180 days
Service account keys: Every 30-90 days
CA keys: Annually (with careful planning)
Emergency rotation: Immediately upon suspected compromise

Rotation process:

# 1. Generate new key pair
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_new

# 2. Add new public key to servers
ssh-copy-id -i ~/.ssh/id_ed25519_new.pub user@server

# 3. Test new key
ssh -i ~/.ssh/id_ed25519_new user@server

# 4. Remove old key from servers
ssh user@server "sed -i '/old_key_fingerprint/d' ~/.ssh/authorized_keys"

# 5. Delete old private key
shred -u ~/.ssh/id_ed25519_old

With SSH CA:

# Rotation is automatic - just issue new certificates
# Old certificates expire naturally

2. Centralization

Centralized key management:

Single source of truth for all keys
Automated discovery and inventory
Consistent policy enforcement
Simplified auditing

Tools and approaches:

SSH CA: Centralized certificate issuance
LDAP/Active Directory: Centralized user management
Vault (HashiCorp): Dynamic SSH credentials
Commercial solutions: CyberArk, BeyondTrust, Teleport

Implementation:

# Example: Vault SSH CA
vault write ssh-client-signer/sign/my-role \
  public_key=@$HOME/.ssh/id_ed25519.pub \
  valid_principals="alice" \
  ttl="8h"

3. Auditing

What to audit:

Key creation and distribution
Access attempts (successful and failed)
Privileged actions performed
Key rotation events
Certificate issuance

Logging configuration:

# /etc/ssh/sshd_config
LogLevel VERBOSE

# Logs to /var/log/auth.log or /var/log/secure

Centralized logging:

# Forward SSH logs to SIEM
# Example: rsyslog configuration
*.* @@siem.example.com:514

Key metrics to track:

Number of active keys per user
Age of keys in use
Failed authentication attempts
Privileged access frequency
Keys without recent usage

Compliance reporting:

Who accessed what, when
Privileged access justification
Key lifecycle documentation
Access review attestations

4. Least Privilege

Principle: Users should have minimum access necessary for their role.

Implementation strategies:

Role-based access:

# Different certificates for different roles
# Developer certificate
ssh-keygen -s ca_key -I "alice-dev" -n alice -V +8h dev_key.pub

# Admin certificate (requires approval)
ssh-keygen -s ca_key -I "alice-admin" -n root -V +1h admin_key.pub

Just-in-time access:

# Request elevated access for specific task
request-access --role admin --duration 1h --reason "Deploy hotfix"

# Access automatically revoked after 1 hour

Command restrictions:

# Limit to specific commands
command="/usr/local/bin/deploy.sh" ssh-rsa AAAAB3...

Bastion hosts:

# All access goes through audited bastion
ssh -J bastion.example.com user@internal-server

5. Secure Storage

Private key protection:

Passphrase protection:

# Always use strong passphrases
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519
Enter passphrase (empty for no passphrase): [strong passphrase]

File permissions:

# Private keys should be readable only by owner
chmod 600 ~/.ssh/id_ed25519

# Public keys can be world-readable
chmod 644 ~/.ssh/id_ed25519.pub

# SSH directory should be restricted
chmod 700 ~/.ssh

Hardware security:

# Use hardware tokens (YubiKey, etc.)
ssh-keygen -t ed25519-sk -O resident -O application=ssh:YubiKey

# Private key never leaves hardware device

CA key protection:

# Store CA private key in HSM
# Never store on general-purpose systems
# Require multiple approvers for use
# Maintain offline backups in secure location

Avoid common mistakes:

# DON'T: Store keys in version control
git add ~/.ssh/id_rsa  # NEVER DO THIS

# DON'T: Email private keys
# DON'T: Store keys in cloud storage without encryption
# DON'T: Use keys without passphrases for interactive use

6. Monitoring and Alerting

Real-time alerts:

Failed authentication attempts
Root access
Access from unusual locations
Certificate issuance
Key rotation failures

Anomaly detection:

Access at unusual times
Access from new IP addresses
Unusual command patterns
Rapid key creation

Example alert rules:

# Alert on 5+ failed attempts in 5 minutes
# Alert on root access outside maintenance windows
# Alert on certificate issuance for privileged roles
# Alert on key age > 180 days

7. Automation

Automate key lifecycle:

Automated key discovery
Scheduled key rotation
Automatic certificate renewal
Orphaned key cleanup

Infrastructure as Code:

# Terraform example
resource "vault_ssh_secret_backend_role" "developer" {
  backend                 = vault_mount.ssh.path
  name                    = "developer"
  key_type                = "ca"
  allowed_users           = "ubuntu,deploy"
  default_user            = "ubuntu"
  ttl                     = "8h"
  max_ttl                 = "24h"
}

CI/CD integration:

# GitLab CI example
deploy:
  script:
    - vault write -field=signed_key ssh/sign/deploy public_key=@$SSH_PUBLIC_KEY > cert
    - ssh -i cert deploy@server "deploy.sh"

Conclusion

SSH key management is critical for enterprise security. Key takeaways: