Page Comparison

Components

• Elastic Compute Cloud (EC2): Amazon’s web-interfaced cloud computing service. Resizable compute capacity. Allows you to utilize your OS to run custom applications. EC2 is scalable, allows you to pay only for what you use, and deploy across different availability zones.

  • The Verdi Nodes are run by EC2

  • At any given time, there are at least 4 EC2 instances

    • GRQ

    • Factotum

    • Metrics

    • Mozart

EC2 instances can be accessed through the AWS Console: Services > Compute > EC2 > Instances

Image Modified

• Autoscaling Groups: An EC2 feature that enables customizable scaling

  • HySDS allows you to specify settings for Autoscaling Groups

Autoscaling Groups can be accessed through the AWS console: Services > Compute > EC2 > Auto Scaling > Auto Scaling Groups

Image Modified

^ Edit out OCO-2 information before making this page public

• Security Groups: Define network settings and firewall on EC2 instances

• Elastic Block Store (EBS): Provides block level storage to be used with EC2 instances, allows you to choose which file system you want.

• Simple Cloud Storage Service (S3): Object store, primarily used to store files.

  • All data products (inputs and outputs) are stored in S3

  • PGE Code is stored in S3 buckets

  • Snapshots stored in S3

• Amazon Machine Image (AMI): Used to create virtual machine within EC2. It contains a snapshot (OS, Application, Application Server), launch permissions, etc. HySDS uses 4 AMIs that correspond to the EC2 instances:

  • GRQ

  • Factotum

  • Metrics

  • Mozart

• Elastic search: AWS allows you to integrate Elasticsearch in order to store metadata

  • All relevant metadata associated with jobs is stored in Elasticsearch. To access elasticsearch in HySDS, open the resource manager and press es:head in the navigation bar

Debugging

• If the autoscaling groups are not working correctly, there is likely a problem with Mozart communication.

Autoscaling Group Structure

Image Modified