Created by Michael D Starch, last modified on Feb 13, 2017

In order to adapt HySDS we need to have a separation of core and adaptation components. Let us start with some definitions:

Core

Core contains the raw data-system gotten from the HySDS team.  This is generally in the form of what is installed via puppet. Core also contains the base python repositories supplied from the HySDS ("osaka", "hysds", "mozart", "figaro", "tosca", "hysds_commons","sciflo" and "user_rules_*")

Adaptation

Comes in three parts:

1. Project created modules such as PGEs, extensions, crawlers, operator tools, etc.

2. Configuration to apply to the core HySDS system

3. "hysds_cluster_setup" fork of the core "hysds_cluster_setup" package. This sets up the overly of the other pieces of adaptation.

The project must fork and maintain their own version of "hysds_cluster_setup", further mentions of this assume the forked version.

Anything in Core should be left as-is and is installed as-is from puppet and from clones directly from the HySDS repositories.

Anything in adaptation should be overlay-ed on-top of HySDS using the hysds_cluster_setup package.

Project Modules and Code

Project modules and code should be created at the discretion of the project. These modules should be placed under "~/mozart/ops/<module-sub-dir>" no other stipulations are made on how those are built nor designed.  However, to deploy these items to the cluster it is strongly recommended that the project use "hysds_cluster_setup" to deploy these resources.

Configuration for HySDS

All configuration for the core-hysds system should live in two places.  The first is "context.sh" under the "hysds_cluster_setup" folder maintained by the project.  "context.sh" contains all the keys, settings, ip addresses, and other basic information on how HySDS is setup.  This then feeds a template system in HySDS cluster fab, and thus fills the necessary configuration for core.  Sometimes, full files are needed and not just replaced values. When a full file is needed, it can be placed under "hysds_cluster_setup/files". Examples are there for the project to modify as they see fit.

On "hysds_cluster_setup"

HySDS cluster setup has several important functions, and it should be used to accomplish these functions.

1. Deposit new core code onto cluster machines

2. Overlay adaptation onto cluster machines

3. Build code bundle for auto-scaled workers

4. Start, Stop, Reset, and Update the cluster

For the purposes of this page, the user is strongly recommended to place all deployment of adaptation and configuration as stated above for the use of "hysds_cluster_setup".  Custom modules can be added to the "cluster.py" to be shipped properly, along with adding custom configuration files to the update_* scripts. In this way the system can be customized and kept consistent across the cluster.