SSH into an EC2 Instance: PEM Keys, Security Groups, and Session Manager

SSH into an EC2 instance with the right PEM key, chmod 400 on it, and the right user (ec2-user for Amazon Linux, ubuntu for Ubuntu). Plus a security group that allows port 22 from your IP. The team that has all four has a working SSH connection in under a minute. The team that misses any of the four is debugging for an hour.

Table of contents

• The right command
• The right security group
• The right PEM key mode
• EC2 Instance Connect
• Session Manager
• FAQ

The right command

The right way to SSH into an EC2 instance:

chmod 400 my-key.pem
ssh -i my-key.pem ec2-user@<public-ip-or-dns>

The right user depends on the AMI. ec2-user for Amazon Linux. ubuntu for Ubuntu. admin for Debian. ec2-user for RHEL.

The right security group

The right security group has an inbound rule for port 22 (SSH) from your IP. The right answer is to restrict the source to your IP (x.x.x.x/32), not 0.0.0.0/0. The wrong answer is to open port 22 to the world — the right answer is to restrict it.

The right PEM key mode

The right file mode for the PEM key is chmod 400. The wrong answer is to leave it at the default 644 — ssh refuses to use a key that is world-readable.

EC2 Instance Connect

The right answer for a one-off browser-based SSH is EC2 Instance Connect. The right workflow:

1. Select the instance in the EC2 console.
2. Click Connect.
3. Choose EC2 Instance Connect tab.
4. Click Connect.

The browser opens a terminal in the EC2 console. The right answer is to use this for quick debugging; the wrong answer is to use it for daily work (use a regular SSH client instead).

Session Manager

The right answer for SSH without opening port 22 is Session Manager. The right setup:

1. Attach the AmazonSSMManagedInstanceCore policy to the instance role.
2. Install the SSM agent (default on most modern AMIs).
3. Open Session Manager in the Systems Manager console.
4. Start a session.

The right answer is that Session Manager does not require an open port 22 and works through the SSM endpoint. The wrong answer is to leave port 22 open and use Session Manager only — the right answer is to close port 22 and use Session Manager exclusively for many use cases.

FAQ

What is the default user for an Amazon Linux instance?

ec2-user. The right answer is to use the AMI-specific user.

What is the default user for an Ubuntu instance?

ubuntu. The right answer is to use the AMI-specific user.

Why does ssh refuse my key?

The most common cause is wrong file mode. The right answer is chmod 400 my-key.pem.

Why is the connection refused?

The right answer is to check the security group inbound rules. The wrong answer is to assume the instance is down.

What is the difference between Instance Connect and Session Manager?

Instance Connect is a browser-based SSH that requires port 22 to be open. Session Manager is an SSM-based session that does not require port 22.

Can I use the same PEM key for multiple instances?

Yes. The right answer is to use the same key for instances in the same environment and different keys for different environments.

What is the right way to manage multiple keys?

Use the SSH config file with IdentityFile per host. The right answer is to set the default key with IdentityFile ~/.ssh/id_ed25519 and override per host.

Can I use a username other than ec2-user?

Yes, but the right answer is to create a new user with sudo useradd -m -G sudo newuser and add your SSH key to ~newuser/.ssh/authorized_keys.

If you are sizing the infrastructure for the kind of project this post covers, the RunxBuild hosting calculator is the right place to model the line items. The compute, the memory, the storage, the bandwidth, the database - each one is a separate number, and the team’s mental model for the platform is the sum of those numbers. The RunxBuild dashboard is where the team sees the actual usage in one place.

Useful related references:

• Sql Managed Instance