We will provision the HySDS component images and then use terraform to deploy them onto cloud vendor-specific IaaS environment.

You should already have provisioned HySDS component instances (mozart, grq, metrics, factotum, and ci) each designated for a particular HySDS component. The suggested minimum CPU and memory requirements for each instance should at least be met (tables below).

Cloud vendor-specific packer scripts can be used to provision the 5 images.

• packer templates for provisioning AWS images

• packer templates for provisioning GCP images

Or, if they are not provisioned to you by an administrator, you can provision them yourself using our puppet modules:

1. mozart - https://github.com/hysds/puppet-mozart

2. metrics - https://github.com/hysds/puppet-metrics

3. grq - https://github.com/hysds/puppet-grq

4. factotum - https://github.com/hysds/puppet-factotum

5. ci - https://github.com/hysds/puppet-cont_int

• Recommended 20 GB of attached general purpose storage for each component's root volume

• The following tables list the CPU and memory requirements for each component instance based on the scope of the HySDS cluster: minimal or optimal:

  
  

  • minimal - Uses instance types that cover the bare minimum CPU and memory requirements for running HySDS in a test or development environment such as this user guide, e.g. Hello World and Hello Dataset

  component	CPU	memory (GB)	instance type
  mozart	2	7	AWS: m5.large; GCP: n1-standard-2; Azure: D2 v3
  metrics	2	7	AWS: m5.large; GCP: n1-standard-2; Azure: D2 v3
  grq	2	7	AWS: m5.large; GCP: n1-standard-2; Azure: D2 v3
  factotum	2	7	AWS: m5.large; GCP: n1-standard-2; Azure: D2 v3
  ci	2	7	AWS: m5.large; GCP: n1-standard-2; Azure: D2 v3

  
  

  • optimal - Uses instance types that cover the CPU and memory requirements for running HySDS in an operational environment or at large scale (500-8000 verdi worker instances)

  component	CPU	memory (GB)	instance type
  mozart	32	208	AWS: r5.2xlarge; GCP: n1-highmem-32; Azure: E32 v3
  metrics	32	208	AWS: r5.xlarge; GCP: n1-highmem-32; Azure: E32 v3
  grq	32	208	AWS: r5.2xlarge; GCP: n1-highmem-32; Azure: E32 v3
  factotum	32	60	AWS: m5.8xlarge; GCP: n1-standard-32; Azure: F32 v2
  ci	32	60	AWS: m5.xlarge; GCP: n1-standard-32; Azure: F32 v2

  
  

• Ensure that the same SSH keypair is used for all 5 of the instances. You should be able to ssh into any of your instances using the PEM key file of this SSH keypair.

• You should already have provisioned 2 buckets from a cloud vendor (e.g. AWS S3, Azure Blob, Google Cloud Storage)

  • one for code and configuration (CODE_BUCKET), e.g. s3://my-code-bucket

  • one for datasets and staging area (DATASET_BUCKET), e.g. s3://my-dataset-bucket

Next Step

Now you can install the HySDS framework software layer onto the systems. Step 2: Install HySDS framework on VM instances