Initial Cluster Setup

For definitions of terminology used, please refer to ourterminology reference.

When installation of the HySDS framework is complete on yourmozartinstance (seeInstallation), we must configure the rest of the cluster instances so that they can talk to each other. We do this using thesdsclicommand on themozartinstance. The idea is that all code and configuration is centralized on themozartinstance and when ready to deploy updates during the development cycle or when upgrading operations, we can push them easily from a single location.

1. Configure your cluster parameters usingsdscli: Thesdsclirepo was installed on yourmozartinstance duringInstallation. Configure your cluster by running:

   cd ~
   source ~/mozart/bin/activate
   sds configure
   

2. Thesds configurecommand will prompt you for your cluster parameters. A description of the parameters with examples is provided below

*** LOOK OVER THE FIELDS AND HAVE THE VALUES READY BEFORE HAND ***

1. Make sure elasticsearch is up on the mozart and grq instances. You can run the following command to check:

   curl 'http://<mozart/grq ip>:9200/?pretty'
   

   you should get answer back from ES, something like this:

   {
    "status" : 200,
    "name" : "Dweller-in-Darkness",
    "cluster_name" : "resource_cluster",
    "version" : {
      "number" : "1.7.3",
      "build_hash" : "05d4530971ef0ea46d0f4fa6ee64dbc8df659682",
      "build_timestamp" : "2015-10-15T09:14:17Z",
      "build_snapshot" : false,
      "lucene_version" : "4.10.4"
    },
    "tagline" : "You Know, for Search"
   }
   
   

   If you can not connect to elastic search, you need to start ElasticSearch in mozart and grq instances:

   sudo systemctl start elasticsearch
   

2. Ensuremozartcomponent can connect to other components over ssh using the configuredKEY_FILENAME. If correctly configured, thesds status allcommand should show that it was able to ssh into each component to check that thesupervisorddaemon was not running like below:

   sds status
   ########################################
   grq
   ########################################
   ----------------------------------------
   third-party services
   ----------------------------------------
   [100.64.134.135] Executing task 'systemctl'
   elasticsearch: ACTIVE
   ----------------------------------------
   supervised services
   ----------------------------------------
   [100.64.134.135] Executing task 'status'
   Supervisord is not running on grq.
   ########################################
   mozart
   ########################################
   ----------------------------------------
   third-party services
   ----------------------------------------
   [100.64.134.172] Executing task 'systemctl'
   rabbitmq-server: ACTIVE
   [100.64.134.172] Executing task 'systemctl'
   redis: ACTIVE
   [100.64.134.172] Executing task 'systemctl'
   elasticsearch: ACTIVE
   ----------------------------------------
   supervised services
   ----------------------------------------
   [100.64.134.172] Executing task 'status'
   Supervisord is not running on mozart.
   ########################################
   metrics
   ########################################
   ----------------------------------------
   third-party services
   ----------------------------------------
   [100.64.134.51] Executing task 'systemctl'
   redis: ACTIVE
   [100.64.134.51] Executing task 'systemctl'
   elasticsearch: ACTIVE
   ----------------------------------------
   supervised services
   ----------------------------------------
   [100.64.134.51] Executing task 'status'
   Supervisord is not running on metrics.
   ########################################
   factotum
   ########################################
   ----------------------------------------
   supervised services
   ----------------------------------------
   [100.64.134.157] Executing task 'status'
   Supervisord is not running on factotum.
   ########################################
   ci
   ########################################
   ----------------------------------------
   third-party services
   ----------------------------------------
   [100.64.134.108] Executing task 'systemctl'
   jenkins: ACTIVE
   ----------------------------------------
   supervised services
   ----------------------------------------
   [100.64.134.108] Executing task 'status'
   Supervisord is not running on ci.
   ########################################
   verdi
   ########################################
   ----------------------------------------
   supervised services
   ----------------------------------------
   [100.64.134.108] Executing task 'status'
   Supervisord is not running on verdi.
   

   Otherwise if any of the components show the following error, for example for the grq component:

   ########################################
   grq
   ########################################
   [100.64.106.214] Executing task 'status'
   
   Fatal error: Needed to prompt for a connection or sudo password (host: 100.64.106.214), but abort-on-prompts was set to True
   
   Aborting.
   Needed to prompt for a connection or sudo password (host: 100.64.106.214), but abort-on-prompts was set to True
   
   Fatal error: One or more hosts failed while executing task 'status'
   
   Aborting.
   One or more hosts failed while executing task 'status'
   

   then there is an issue with the configuredKEY_FILENAMEonmozartor theauthorized_keysfile under the component's~/.sshdirectory for userOPS_USER. Resolve this issue before continuing on.

3. Update all HySDS components:

   sds update all
   

   If you receive any errors, they will need to be addressed.

4. Start up all HySDS components:

   sds start all
   

5. View status of HySDS components and services:

   sds status all
   

6. During installation, the latest versions of thelightweight-jobscore HySDS package and theverdidocker image was downloaded. Next we import thelightweight-jobspackage:

   cd ~/mozart/pkgs
   sds pkg import container-hysds_lightweight-jobs.*.sdspkg.tar
   

7. Finally we copy theverdidocker image to the code bucket (CODE_BUCKETas specified duringsds configure). EnsureVERDI_PRIMER_IMAGEurl is consistent:

   aws s3 cp hysds-verdi-latest.tar.gz s3://<CODE_BUCKET>/hysds-verdi-latest.tar.gz
   

Next Steps

Now that you have your HySDS cluster configured, continue on toHello World.

To configure Autoscaling groups for your HySDS cluster, continue on toCreate-AWS-Autoscaling-Group-for-Verdi.

To configure a staging area for your HySDS cluster, continue on toCreate-AWS-Resources-for-Staging-Area.