Access to an Environment via SSM
Overview
Due to all our data living in secure environments you cannot directly connect to Database or OpenSearch endpoints from your local machine.
Therefore should you need to connect directly to one of these services (to run adhoc queries, dig into weird data, or monitor database query performance)
it is necessary to connect to some running compute inside the VPC where it is possble to connect to these services.
This was formerly achieved via an AWS Service called Cloud9, however that service has been deprecated by AWS, and left a potentially exploitable attack vector into our services, which we wanted to close off.
Starting the Operator EC2 instance
There is now a shared operator instance for each environment, multiple people can use it at the same time without any issues, but you might see local files popping up and changing if you do as it’s all shared at the operator level. It is a switched off by default (to save money) so you need to start it up and shut it down when you’re finished using it. If someone else is using it and it’s already started up shout out in #opg-sirius-develop to let people know not to shut it down once they’re finished because you might still be using it, and last person to finish with it is responsible for turning it off.
- Using a browser log into the AWS Console and switch to the Operator role in the account you want to access an environment in.
- In the Search box at the top of the console type ‘EC2’ and switch to that service page.
- Click on ‘Instances’ on the side bar to the left of the main EC2 console.
- You will see a number of instances named ‘operator-{environment_name}’, right click on the one that matches up to the environment that you want to connect to.
- Click ‘Start instance’, give it 40-60 seconds to start up and provision itself (you don’t need to wait for the Status checks to pass, but you can if you like).
- Right click on the instance and select ‘Connect’.
- Make sure ‘Session Manager’ is highlighted at the connection method, and that session manager is registered and available (you will need to refresh the page if it’s not).
- Click ‘Connect’
- You will be presented with a terminal window, switch to the root user with
sudo su -(the hyphen is important) and you’ll be presented with some help text which is detailed below.
Using the environment
Connecting to PostgreSQL
As the help text says, simply type psql to connect to the default read/write instance of the Cluster for that environment.
Should you want to connect to the reader instances (usually for reports or query monitoring) type psql --host $PGROHOST
OpenSearch
The OpenSearch connection string is stored in the environment under a variable called $SEARCH so you can run curl $SEARCH/_cluster/health?pretty to connect
File Transfer
As there is no longer a GUI that lets you transfer files to and from the remote instance’s local storage it needs to be done via S3 Bucket.
Each environmnet has a regional S3 bucket specifically for this purpose called s3-downloads.{environment_name}.{region}.sirius.opg.justice.gov.uk and you’ll find that
under the Bucket list in S3 in the AWS console (Search bar -> S3 -> Buckets). There is a directory in that bucket called ssm-files to keep these apart from other files
that may reside in that bucket.
You can drag and drop files here from your machine, when doing so expand the ‘Properties’ section in the upload screen and make sure you select ‘Specify an encryption key’ or the upload will fail.
On the instance you can use the AWS CLI tool to pull files from and put files in the bucket.
There are two helper functions copytos3 and copyfroms3 they do exactly what they sound like.
copytos3 my-file.csv will look for a file called my-file.csv in the current directory and copy it to the ‘ssm-files’ directory in the bucket.
copyfroms3 my-bucket-file.sql will copy a file called my-bucket-file.sql from the ‘ssm-files’ directory in the Bucket to the current directory.
Docker
Docker is installed by default and you can pull in containers to help you with tasks if you need to. You can also log into our ECR Registry and pull App Images down if you need to.
Things to be aware of
Persistence
The drives of these instances are ephemeral, so once shut down they lose all data, so don’t leave files you need on the instance between uses.
Network Egress
These instances sit in the Private/App Layer of our network structure so if you interact with external services you will be using the same Network egress routes as our applications. This means you can connect to/test services that are IP Restricted to our VPC Egress IP Addresses.
When you’re finished
Once you’re finished with the environment.
- Ctrl+D until the session manager exits or close the browser tab.
- Make sure no one else is using the instance, then right click it in the ‘Instances’ list and choose ‘Stop Instance’, and confirm.
If the environment is on for longer than three hours at a time then whomever is on PagerDuty will be notified and you may be asked if you are using it, or if you forgot to turn it off :D