Hook Hua edited this page on Jun 12, 2018

Installation

For definitions of terminology used, please refer to our terminology reference.

Prerequisites

We will provision the HySDS component images and then use terraform to deploy them onto cloud vendor-specific IaaS environment.

You should already have provisioned HySDS component instances (mozart, grq, metrics, factotum, and ci) each designated for a particular HySDS component. The suggestedminimumCPU and memory requirements for each instance should at least be met (tables below).

Cloud vendor-specificpackerscripts can be used to provision the 5 images.

• packertemplates for provisioning AWS images

• packertemplates for provisioning GCP images

Or, if they are not provisioned to you by an administrator, you can provision them yourself using ourpuppetmodules:

1. mozart-https://github.com/hysds/puppet-mozart

2. metrics-https://github.com/hysds/puppet-metrics

3. grq-https://github.com/hysds/puppet-grq

4. factotum-https://github.com/hysds/puppet-factotum

5. ci-https://github.com/hysds/puppet-cont_int

• The following tables list the CPU and memory requirements for each component instance based on the scope of the HySDS cluster:minimum,development,operations:

  • minimum- Uses instance types that cover the bare minimum CPU and memory requirements for running HySDS in a test environment such as this user guide, e.g.Hello WorldandHello Dataset

  • development- Uses instance types that cover the CPU and memory requirements for running HySDS in a development environment

  • optimal- Uses instance types that cover the CPU and memory requirements for running HySDS in an operational environment or at large scale (500-8000verdiworker instances)

  
  
  

• Ensure that the same SSH keypair is used for all 5 of the instances. You should be able to ssh into any of your instances using the PEM key file of this SSH keypair.

• You should already have provisioned 2 buckets from a cloud vendor (e.g. AWS S3, Azure Blob, Google Cloud Storage)

  • one for code and configuration (CODE_BUCKET), e.g. s3://my-code-bucket

  • one for datasets and staging area (DATASET_BUCKET), e.g. s3://my-dataset-bucket

Install HySDS framework on each instance

The latest releases are here:https://github.com/hysds/hysds-framework/releases. Each was taken from the latest head state of all repos at time of release.

Install HySDS framework on mozart

1. Log into yourmozartinstance

   ssh -i <PEM file> ops@<mozart IP>

2. Copy the PEM file that you used to log into themozartinstance into ~/.ssh directory. You can use scp to transfer the file:

   scp -i <PEM file> <PEM file> ops@<mozart IP>:~/.ssh

3. Ensure the permissions on the PEM file are readable by your account only:

   chmod 400 <PEM file>

4. If there's amozartdirectory already under you home directory, move or remove it:

5. Clone the HySDS framework repository and enter it

6. Select the HySDS framework release tag you'd like to install for mozart

7. Install the HySDS release you want to install (e.g. v2.1.0-beta.1) for the mozart component

   You could also install the development version which pulls the master branch of each HySDS repo:

Next Steps

Now that you have the HySDS framework installed on your cluster's mozart instance, continue on to Cluster Setup.