When installation of the HySDS framework is complete on your mozart instance (see Installation), we must configure the rest of the cluster instances so that they can talk to each other. We do this using the sdscli command on the mozart instance. The idea is that all code and configuration is centralized on the mozart instance and when ready to deploy updates during the development cycle or when upgrading operations, we can push them easily from a single location.

1. Configure your cluster parameters using sdscli: The sdscli repo was installed on your mozart instance during Installation. Configure your cluster by running:

   cd ~
   source ~/mozart/bin/activate
   sds configure

2. The sds configure command will prompt you for your cluster parameters. A description of the parameters with examples is provided below

*** LOOK OVER THE FIELDS AND HAVE THE VALUES READY BEFORE HAND ***

QUEUES:
  - QUEUE_NAME: dumby-job_worker-small
    INSTANCE_TYPES:
      - t2.medium
      - t3a.medium
      - t3.medium
  - QUEUE_NAME: dumby-job_worker-large
    INSTANCE_TYPES:
      - t2.medium
      - t3a.medium
      - t3.medium

1. Make sure elasticsearch is up on the mozart and grq instances. You can run the following command to check:

   curl 'http://<mozart/grq ip>:9200/?pretty'

   
   

   you should get answer back from ES, something like this:

   
   

   {
    "status" : 200,
    "name" : "Dweller-in-Darkness",
    "cluster_name" : "resource_cluster",
    "version" : {
      "number" : "1.7.3",
      "build_hash" : "05d4530971ef0ea46d0f4fa6ee64dbc8df659682",
      "build_timestamp" : "2015-10-15T09:14:17Z",
      "build_snapshot" : false,
      "lucene_version" : "4.10.4"
    },
    "tagline" : "You Know, for Search"
   }
   
   

   If you can not connect to elastic search, you need to start ElasticSearch in mozart and grq instances:

   
   

   sudo systemctl start elasticsearch

2. Ensuremozartcomponent can connect to other components over ssh using the configuredKEY_FILENAME. If correctly configured, thesds status allcommand should show that it was able to ssh into each component to check that thesupervisorddaemon was not running like below:

   sds status all
   ########################################
   grq
   ########################################
   [100.64.106.214] Executing task 'status'
   Supervisord is not running on grq.
   ########################################
   mozart
   ########################################
   [100.64.106.38] Executing task 'status'
   Supervisord is not running on mozart.
   ########################################
   metrics
   ########################################
   [100.64.106.140] Executing task 'status'
   Supervisord is not running on metrics.
   ########################################
   factotum
   ########################################
   [100.64.106.64] Executing task 'status'
   Supervisord is not running on factotum.
   ########################################
   ci
   ########################################
   [100.64.106.220] Executing task 'status'
   Supervisord is not running on ci.
   ########################################
   verdi
   ########################################
   [100.64.106.220] Executing task 'status'
   Supervisord is not running on verdi.

   Otherwise if any of the components show the following error, for example for the grq component:

   ########################################
   grq
   ########################################
   [100.64.106.214] Executing task 'status'
   
   Fatal error: Needed to prompt for a connection or sudo password (host: 100.64.106.214), but abort-on-prompts was set to True
   
   Aborting.
   Needed to prompt for a connection or sudo password (host: 100.64.106.214), but abort-on-prompts was set to True
   
   Fatal error: One or more hosts failed while executing task 'status'
   
   Aborting.
   One or more hosts failed while executing task 'status'

   then there is an issue with the configuredKEY_FILENAMEonmozartor theauthorized_keysfile under the component's~/.sshdirectory for userOPS_USER. Resolve this issue before continuing on.

3. Update all HySDS components:

   sds update all

   If you receive any errors, they will need to be addressed.

4. Start up all HySDS components:

   sds start all

5. View status of HySDS components and services:

   sds status all

6. During installation, the latest versions of thelightweight-jobscore HySDS package and theverdidocker image was downloaded. Next we import thelightweight-jobspackage:

   cd ~/mozart/pkgs

   
   

   sds pkg import container-hysds_lightweight-jobs.*.sdspkg.tar

7. Finally we copy theverdidocker image to the code bucket (CODE_BUCKETas specified duringsds configure). EnsureVERDI_PRIMER_IMAGEurl is consistent:

   aws s3 cp hysds-verdi-latest.tar.gz s3://<CODE_BUCKET>/hysds-verdi-latest.tar.gz

Next Step

Now that you have your HySDS cluster configured, continue on to Step 5: Running your First "Hello World" Job