Introduction

This page outlines some of the key concepts of HySDS, and links to related resources to use while onboarding with HySDS.

What is HySDS?

HySDS is a custom hybrid compute system. It is a science data system designed to automate bulk processing of large volumes of data.

HySDS supports various NASA projects, including: Advanced Rapid Imaging and Analysis (ARIA), Surface Water and Ocean Topography (SWOT), Soil Moisture Active Passive (SMAP), and NASA-ISRO SAR Mission (NISAR).

HySDS can run on a variety of compute resources, including Amazon AWS, Microsoft Azure, Google Cloud, and NASA’s Pleiades supercomputer.

Terminology & Diagrams

Onboarding Reference Materials

Glossary of Common HySDS Terms

Repository of HySDS Diagrams

System-wide Component Overview

HySDS Core “Bird’s-Eye” Diagram

HySDS Technology Stack

Open

HySDS Level 1 Component Overview

HySDS Level 2 Component Overview

Glossary of HySDS specific & Third Party Components

Initial Concepts (Step 1)

Tosca

The HySDS GUI for submitting jobs for data processing

• Intro to Tosca: the HySDS GUI

Figaro

The HySDS GUI for tracking and managing submitted jobs

• Intro to Figaro: the HySDS Resource Manager

Job Faceting

Functionality enabling targeted searches based on job parameters

• Share Links of Faceted Content

On-Demand Jobs

One-time job processing options for bulk data processing

• Submit an On-Demand Job in Facet Search

Trigger Rules

Automated job processing based on custom pre-configured conditions

• Create, Edit, & Delete Trigger Rules

Lightweight Jobs

Lightweight jobs are common across all HySDS adaptations. They allow easy bulk processing of data via frequently used job tasks.

• Lightweight Jobs in Resource Manager

Intermediate Concepts (Step 2)

RabbitMQ

Third-party software managing HySDS job queues. Operators utilize RabbitMQ for troubleshooting; it serves as an ultimate source of truth for job status.

• RabbitMQ Guide

• RabbitMQ Utilities

Mozart

• Mozart-Resource Management

Additional Concepts (Step 3)

Elasticsearch (ES)

Repository for much of the HySDS state information

• Navigating the Elasticsearch Plug-in head Database

• https://wiki.jpl.nasa.gov/display/S6MS/Loading+Data+Into+ElasticSearch

• https://wiki.jpl.nasa.gov/display/NISARSDS/Elasticsearch+Service

• Elasticsearch Upgrade: 1.7 to 7.1

Amazon Web Services (AWS)

Regions & Availability Zones (AZ)

Regions are geographically distinct groupings of AWS resources. Most AWS resources are region-specific, including S3, EC2, and ASG. Availability zones consist of a physical data center; each region has at least 3 availability zones.

Simple Storage Service (S3)

HySDS uses AWS S3 object storage for its data storage requirements.

EC2 Instances

Verdi and Factotum worker nodes use EC2 instances to provide scalable computing resources. Amazon offers on-demand, spot, and reserved pricing options.

Spot Instance Pricing

HySDS uses both on-demand and spot instances.

AWS Auto Scaling Groups (ASG)

Auto Scaling Groups are resizable clusters of EC2 instances used for large-scale data processing.

Amazon Machine Images (AMIs)

HySDS uses customized AMIs that are shared across JPL missions. Experienced HySDS users should confirm AMI compatibility with the configuration of HySDS their using.

CloudWatch

• HySDS monitors Mozart metrics via CloudWatch

• Controls scaling behavior of the HySDS ASGs

Boto3

Python library that allows you to programmatically communicate with AWS. HySDS devs use this

AWS CLI

Allows users to interact with Amazon via the command line.

Advanced Concepts (Step 4)

The main components: GRQ, Factotum, Docker

• HySDS Level 1 Overview

• HySDS Level 2 Overview

Docker

HySDS uses Docker containers to run Product Generating Executor’s (PGE’s). An introduction to Docker.

Role Specific Concepts (Step 5)

Depends on the actual role, for example:

• Software dev → Docker/PGE

• Ops → How to SSH into the machines