Components
Elastic Compute Cloud (EC2): Amazon’s web-interfaced cloud computing service. Resizable compute capacity. Allows you to utilize your OS to run custom applications. EC2 is scalable, allows you to pay only for what you use, and deploy across different availability zones.
The Verdi Nodes are run by EC2
At any given time, there are at least 4 EC2 instances
GRQ
Factotum
Metrics
Mozart
EC2 instances can be accessed through the AWS Console: Services > Compute > EC2 > Instances
Autoscaling Groups: An EC2 feature that enables customizable scaling
HySDS allows you to specify settings for Autoscaling Groups
Autoscaling Groups can be accessed through the AWS console: Services > Compute > EC2 > Auto Scaling > Auto Scaling Groups
^ Edit out OCO-2 information before making this page public
Security Groups: Define network settings and firewall on EC2 instances
Elastic Block Store (EBS): Provides block level storage to be used with EC2 instances, allows you to choose which file system you want.
Simple Cloud Storage Service (S3): Object store, primarily used to store files.
All data products (inputs and outputs) are stored in S3
PGE Code is stored in S3 buckets
Snapshots stored in S3
Amazon Machine Image (AMI): Used to create virtual machine within EC2. It contains a snapshot (OS, Application, Application Server), launch permissions, etc. HySDS uses 4 AMIs that correspond to the EC2 instances:
GRQ
Factotum
Metrics
Mozart
Elastic search: AWS allows you to integrate Elasticsearch in order to store metadata
All relevant metadata associated with jobs is stored in Elasticsearch. To access elasticsearch in HySDS, open the resource manager and press es:head in the navigation bar
Debugging
If the autoscaling groups are not working correctly, there is likely a problem with Mozart communication.