SSM Sessions Manager on Steroids

SSM Sessions Manager is a managed service provided by AWS allowing remote access to managed EC2 instances running within — or outside of — an AWS VPC, and whether or not they are publicly accessible. It is the AWS recommended service for remotely accessing private EC2 or database instances from a local development machine, without having to rely on Bastion Hosts exposed to the public Internet.

In this article, we will walk through how SSM Sessions Manager works, understand how we can leverage its abilities, and explore how to build on top of it in order to extend its usage within a productive and secure development workflow.

Terminal by Lukas on Unsplash

Accessing Private Instances

The most common use-case enabled by SSM Sessions Manager is to allow engineers to gain secure access to EC2 instances located in a private subnet from a remote development machine that sits outside of the AWS network.

Behind the scenes, the AWS SSM service proxies the connection from the local machine to the SSM managed instance placed in a VPC through a SigV4 signed WebSocket connection, and spawns a shell associated with a specific user — typically ssm-user on Linux.

💡 — You can think of Sessions Manager as an SSH equivalent without having to manage asymmetric keys, exposing instances to the public Internet, and with observability and access control built-in.

# An example initiating an SSM connection to a private EC2 instance.
➜ aws ssm start-session --target i-exampleid
Starting session with SessionId: ubuntu@domain.com-exampleid
$ whoami
ssm-user

Forwarding Ports to Instance Applications

In a typical development workflow, developers want to remotely access specific applications running on a remote instance — such as a web server — for testing or development purposes. Sessions Manager makes it possible to specify the port of a listening application on a remote instance and forward it to a port on a local machine.

Below is an example forwarding a web server listening on port 8080 on an EC2 instance, to the port 8081 on a local machine.

$ aws ssm start-session \
  --target i-exampleid \
  --document-name AWS-StartPortForwardingSession \
  --parameters '{"portNumber":["8080"],"localPortNumber":["8081"]}'

💡 — One of the nice things about SSM port forwarding is that the remote instance is not required to have any open ports in its Security Group. In fact, it works even if the remote application listens on the localhost interface, thus being totally invisible from the outside of the instance.

Forwarding Ports to other Instances

Forwarding a remote port from an EC2 instance to a local port on a development machine is very useful, but what about forwarding ports from other hosts within the VPC — such as to an RDS database?

We commonly name instances used to access other instances in their network jump hosts, and Sessions Manager provides us with the ability to use a target instance as a jump host to other hosts in the VPC. In the below example, we use a jump host to locally forward the port of an RDS database located in the same VPC as the jump host.

$ aws ssm start-session \
  --target i-exampleid \
  --document-name AWS-StartPortForwardingSessionToRemoteHost \
  --parameters '{"host":["mydb.us-east-2.rds.amazonaws.com"],"portNumber":["3306"], "localPortNumber":["3306"]}'

This is similar to the way SSH Remote Port Forwarding works, with the added benefit of the jump host not being exposed to the Internet and with the monitoring and observability provided by the SSM service that logs all port forwarding action.

Important — You should always ensure AWS’s security constructs such as IAM Policies and Security Groups are correctly used to limit access between hosts in your VPC, even within the boundaries of private subnets, to mitigate the risk for lateral movement.

Existing Gaps with Sessions Manager

The SSM Sessions Manager plugin provides very useful features that not only enhances developer productivity but also greatly improves security overall as it removes the need for bastion hosts and instances reachable over the public Internet.

However, there are still remaining aspects of the end-to-end developer workflow that are not entirely covered by AWS Sessions Manager and that we can further improve :

• The lack of support of UDP based applications — at this time, only TCP port forwarding is supported.
• The lack of support of the Sessions Manager protocol by most third-party applications that traditionally support SSH for accessing remote Linux machines.
• The inability to address an EC2 instance using its hostname, DNS name, or private IP address. At this time, only instance identifiers are supported which can be challenging to remember, especially when dealing with multiple instances.

Those gaps can be mitigated by augmenting the AWS Sessions Manager Plugin with other tools. In the next sections, we will explore how those integrations can help developers further improve their experience and productivity on AWS.

Sessions Manager + OpenSSH

The first integration we’ll discuss is with OpenSSH itself which can be integrated with the Sessions Manager Plugin in order to enable developers to run regular SSH sessions through a Sessions Manager tunnel.

The integration with OpenSSH involves specifying a specific configuration, which on Linux and MacOS is typically defined in the ~/.ssh/config file. The ProxyCommand directive from OpenSSH will be used for the integration with Sessions Manager. OpenSSH describes this directive in the following terms :

ProxyCommand specifies the command to use to connect to the SSH server. […] The command can be basically anything, and should read from its standard input and write to its standard output. It should eventually connect an sshd(8) server running on some machine, or execute sshd -i somewhere.

Below is an example configuration which leverages the AWS-StartSSHSession document provided by SSM to integrate with applications such as OpenSSH.

Sample configuration integrating OpenSSH with SSM

This configuration indicates to OpenSSH that for each given hostname matching the patterns i-* or mi-*, it should spawn the Sessions Manager plugin to establish a connection to the EC2 instance, and then write its byte stream on the standard input of the Sessions Manager process. The configuration also forwards SSH arguments such as the hostname and the port to the Sessions Manager plugin, resulting in a seamless experience for the user or the application using the OpenSSH interface.

With this configuration enabled, it is now possible to connect to a private EC2 instance with OpenSSH using the identifier of the instance as the host name and the private key associated with the instance.

$ ssh -i /path/to/cert.pem ec2-user@i-exampleid

This allows the integration of SSH-aware applications (such as Ansible and Packer) with Sessions Manager. It also enables the use of OpenSSH’s native port-forwarding capabilities to forward TCP and UDP ports over the SSM tunnel. However, we still lack a way to specify a friendly name for our instances when connecting to them.

This can be easily fixed by extending our OpenSSH configuration to use the AWS CLI to resolve the name of the instance at connection time. To do that, you can first specify a friendly name for your EC2 instance using the AWS console as depicted below.

Providing an EC2 Instance with a friendly name

AWS will associate a new tag to the instance with a key of Name and a value of aws-awesome-instance which we can resolve at connection time by updating our initial OpenSSH configuration. Below is the refactored OpenSSH configuration file that delegates the instance identifier resolution process to a Bash script, and specifies the default SSH private key to use.

Note the added pattern which will match any host name starting with aws-. This is of course an example, feel free to adapt it to match your own naming conventions.

Configuration supporting Instance Identifiers and Friendly Names

Below is the content of the initiate-ssm-connection.sh script which will resolve the user-provided host name to an EC2 instance identifier before creating the SSM tunnel.

This new configuration makes it possible to connect to the instance using its name, in addition to its identifier.

$ ssh user@aws-awesome-instance
Welcome to Ubuntu 22.04.2 LTS (GNU/Linux 5.15.0-1031-aws x86_64)
ubuntu@ip-172-31-17-148:~$

Sessions Manager + sshuttle

In this last section, we are going to build on top of the OpenSSH integration we saw in the previous section to further enhance our SSM workflow, and use a truly remarkable tool called sshuttle.

“A Transparent proxy server that works as a poor man’s VPN.”

In a nutshell, sshuttle works by establishing an SSH connection to a remote host and routes the traffic from a local machine targeting a specific IP CIDR to a remote network such as, in our case, an AWS VPC.

To forward local network sessions and be able to address hosts within a VPC from my local machine, I created a new VPC with a CIDR of 172.31.0.0/16 which I will use as a reference. To establish a local route from my computer to the remote VPC, I can simply invoke sshuttle as follow :

$ sshuttle -r user@aws-awesome-instance 172.31.0.0/16

This instructs sshuttle to initiate an SSH-over-SSM connection — remember we’re still using SSM tunneling as per our configuration in the previous section — to the EC2 instance aws-awesome-instance.

Illustration of the sshuttle network flow. Credits to AWS.

When the connection is established, sshuttle will route all traffic targeting the 172.31.0.0/16 CIDR through the created tunnel, mimicking the behavior of a split-tunneling VPN.

I have created an RDS database within the VPC, and I can now seamlessly establish a connection to it without relying on a complex port forwarding logic.

$ mysql -h 172.31.4.135 -u admin -p

It is also possible to request sshuttle to forward DNS queries made on the local development machine to the AWS VPC, thus benefiting from in-VPC name resolution and being able to address hosts in the VPC using their private DNS name. To do so, you can pass the --dnsoption to sshuttle.

$ sshuttle --dns -r user@aws-awesome-instance 172.31.0.0/16
$ mysql -h db.instance.internal.compute -u admin -p

Sshuttle will now intercept all DNS traffic originating from the local machine and have it resolved by the jump host in the VPC.

Note that in some situations, this might not be ideal as we will lose the ability to issue DNS queries locally to our default DNS server(s), and will not be able to resolve public domain names anymore if the jump host is isolated from the Internet.

Bonus — Split DNS with sshuttle

The solution is to split the DNS resolution queries between the local DNS resolver and the jump host’s resolver. To do so, we need additional configuration on the local host.

In an AWS VPC, the default Private DNS Resolver’s IP address can be deducted from the VPC CIDR, and is located at the VPC+2 IP address. For our VPC which has a CIDR of 172.31.0.0/16, this means that its DNS resolver will be available at 172.31.0.2. We can take advantage of this information to provide OS-specific configuration to split the DNS traffic between the local DNS resolver and the VPC resolver.

MacOS

On MacOS we can specify the domain names we want to see resolved by the VPC DNS resolver by adding an entry in /etc/resolver.

# Resolve *.compute.internal (EC2) names remotely.
$ sudo bash -c 'echo nameserver 172.31.0.2' > /etc/resolver/compute.internal

# Ask sshuttle to forward DNS queries to 172.31.0.2 remotely.
$ sshuttle --ns-hosts=172.31.0.2 -r user@aws-awesome-instance 172.31.0.0/16

This will allow you to only resolve EC2 internal domains by the VPC DNS resolver. You can add additional domains according to your needs.

Linux

For Linux, it’s more complex as the default resolv.conf does not allow to specify domain-specific DNS servers. You will need to use a local DNS cache resolver such as dnsmasq to point specific domain to the VPC domain resolver. A technical explanation of how to do so is available here.

Conclusion

We have explored in this article different improvements that can be built on top of the AWS SSM Sessions Manager service, that further streamlines the developer experience and productivity.

To conclude on the conducted research and to compare the different proposed improvements, I created a feature comparison table that summarizes the different approaches discussed in this article associated with their own features.

Check out the ssm-supercharged repository for an implementation of how to streamline your SSM workflow with OpenSSH, SSM, EC2 Instance Connect and sshuttle!