In 2019, Netcraft found 74.2% of web-facing machines run Linux. During an IPv4-wide census in 2016, an OpenSSH banner was detected 75% of the time when there was a response on TCP port 22. It's safe to say OpenSSH is probably the world's most popular software for connecting to servers remotely. It's also one of the most prized attack vectors given the functionality offered to anyone able to connect.
Hardening the security aspects of an OpenSSH configuration is very challenging. It's even worse for teams that aren't focused on network security and can't justify the budget for consultants setting up bespoke systems.
Even for security-minded teams, the following issues can come up:
- Having OpenSSH listen on odd port numbers can be defeated by port scanning.
- If an employee can connect via their mobile phone or laptop and said device is sold, stolen or seized then any port-knocking configuration, as well as any private keys, will be compromised.
- Some firms use IPv4 filtering to only allow for connectivity from their office but then go on to improperly secure their office Wi-Fi. The work-from-home revolution adds complexity to this setup.
- ~/.ssh/authorized_keys files are rarely reviewed. Staff depart a business but their credentials can live on.
- If the SSH keys on your laptop are ever corrupted and weren't backed up, you'll be locked out.
- Businesses can lose access to remote systems because any staff that did have access left the business and destroyed their employer-specific keys.
- I've seen businesses audit their EC2 instances, see machines they don't recognise and then struggle to find anyone that has access. They can't then drop a new SSH key into the ~/.ssh/authorized_keys file and connect to the machine to see what it's running.
SSH keys are effectively radioactive. They are very useful but there are much safer and more reliable ways to connect via SSH.
In this post, I'll walk through connecting to an EC2 instance using my Google account, complete with two-factor authentication. I'll use BastionZero, a service built by a Boston-based startup of the same name. BastionZero was founded by Ethan Heilman and fellow Canadian Sharon Goldberg.
BastionZero allows you to connect to your servers using a variety of two-factor authentication services. You can connect from anywhere without having to set up VPNs or jump boxes. Connection auditing and SSH command logging are enabled by default.
Launching an EC2 Instance
For this exercise, I'll launch a $3.80 / month t3.nano instance in AWS' eu-west-1 region in Ireland running Ubuntu 22.04 LTS. It has 512 MB of RAM, 2 vCPUs, 5 Gbps network connectivity and 8 GB of gp2 storage capacity has been allocated for the root partition.
I generated an SSH key via the EC2 Web UI and stored it in ~/.ssh/my.pem. Below is my ~/.ssh/config file.
$ cat ~/.ssh/config
ServerAliveInterval 50
Host testbz
HostName 3.252.38.36
User ubuntu
IdentityFile ~/.ssh/my.pem
I'll ensure the permissions on my key are tight enough and then run uptime on the EC2 instance to ensure I can connect without issue.
$ sudo chmod 600 ~/.ssh/my.pem
$ ssh testbz uptime
10:10:22 up 4 min, 0 users, load average: 0.11, 0.15, 0.08
Launching BastionZero's Agent
I've logged into BastionZero's Web UI with my Google account. On the main dashboard, I've clicked the yellow "Create" button in the top right and selected "API Key" from the drop-down menu. I'll set the name of the key to "testbz" and tick the registration key box. I'm then presented with a Client ID and Client Secret.
I'll SSH into the EC2 instance and install BastionZero's Agent. The source code for this agent, which is made up of ~13K lines of GoLang, is available on GitHub.
$ ssh testbz
$ sudo apt-key adv \
--keyserver keyserver.ubuntu.com \
--recv-keys E5C358E613982017
$ sudo add-apt-repository \
'deb https://download-apt.bastionzero.com/production/apt-repo stable main'
$ sudo apt update
$ sudo apt install bzero
$ sudo bzero -registrationKey *registration API key secret*
I can now see that the agent is not listening on any ports but instead connecting out to BastionZero's command-and-control server.
$ sudo lsof -OnP | grep LISTEN
systemd-r 392 systemd-resolve 14u IPv4 16546 0t0 TCP 127.0.0.53:53 (LISTEN)
sshd 683 root 3u IPv4 18270 0t0 TCP *:22 (LISTEN)
sshd 683 root 4u IPv6 18281 0t0 TCP *:22 (LISTEN)
$ sudo lsof -OnP | grep bzero | grep TCP
bzero 2408 root 7u IPv4 28283 0t0 TCP 172.30.2.242:36892->13.224.68.18:443 (ESTABLISHED)
bzero 2408 2409 bzero root 7u IPv4 28283 0t0 TCP 172.30.2.242:36892->13.224.68.18:443 (ESTABLISHED)
bzero 2408 2410 bzero root 7u IPv4 28283 0t0 TCP 172.30.2.242:36892->13.224.68.18:443 (ESTABLISHED)
bzero 2408 2411 bzero root 7u IPv4 28283 0t0 TCP 172.30.2.242:36892->13.224.68.18:443 (ESTABLISHED)
bzero 2408 2536 bzero root 7u IPv4 28283 0t0 TCP 172.30.2.242:36892->13.224.68.18:443 (ESTABLISHED)
bzero 2408 2537 bzero root 7u IPv4 28283 0t0 TCP 172.30.2.242:36892->13.224.68.18:443 (ESTABLISHED)
Their server is also located in the same AWS Region as my EC2 instance.
$ curl ipinfo.io/13.224.68.18
{
"ip": "13.224.68.18",
"hostname": "server-13-224-68-18.dub2.r.cloudfront.net",
"city": "Dublin",
"region": "Leinster",
"country": "IE",
"loc": "53.3331,-6.2489",
"org": "AS16509 Amazon.com, Inc.",
"postal": "D02",
"timezone": "Europe/Dublin",
"readme": "https://ipinfo.io/missingauth"
}
Connecting via BastionZero's Client
I've now removed the ability to connect via SSH to the EC2 instance via AWS' EC2 Security Group Web UI.
BastionZero's macOS client can be installed via Homebrew. It's made up of 14,595 lines of TypeScript and is maintained on GitHub.
$ brew install bastionzero/tap/zli
I'll run the login command which will bring up my browser and allow me to authenticate via my Google account.
$ zli login
Once logged in the following is printed to the CLI.
Login required, opening browser
Login successful
Logged in as: ..., bzero-id: ..., session-id: ...
I'll list out the targets I have access to. Below I can see the EC2 instance I set up listed.
$ zli list-targets
┌───────┬─────────────────┬────────────────┐
│ Type │ Name │ Environment │
├───────┼─────────────────┼────────────────┤
│ Bzero │ ip-172-30-2-242 │ Default │
└───────┴─────────────────┴────────────────┘
I can now connect via BastionZero's Client.
$ zli connect ip-172-30-2-242
$ uptime
10:29:23 up 23 min, 1 user, load average: 0.00, 0.00, 0.00