This is the version 1.0 (2019-05-17) Software Design Document (SDD) for the Hybrid Cloud Science Data System (HySDS).

Introduction

Purpose

This document provides the high level design information to understand the architecture and design of the core HySDS framework. It is not intended to replace reading the code for details.

Background

The primary responsibility of a science data system (SDS) is to ingest science data as input and process them into higher level science data products as output.

...

...

Syntax

{
  "command": "string",
  "recommended-queues": [ "string" ],
  "disk_usage":"\d+(GB|MB|KB)",
  "imported_worker_files": {
    "string": "string",
    "string": [ "string" ],
    "string": [ "string", "ro" | "rw" ]
  },
  "dependency_images": [
    {
      "container_image_name": "string",
      "container_image_url": "string" | null,
      "container_mappings": {
        "string": "string",
        "string": [ "string" ],
        "string": [ "string", "ro" | "rw" ]
      }
    }
  ],
  "soft_time_limit": int,
  "time_limit": int,
  "disable_pre_builtins": true | false,
  "pre": [ "string" ],
  "disable_post_builtins": true | false,
  "post": [ "string" ],
  "params": [
    {
      "name": "string",
      "destination": "positional" | "localize" | "context"
    }
  ]
}

...

...

...

...

1. Look up the RangeBeginning and RangeEnding times to give to the L0B PGEs,
2. Find all L0A_Prime files that overlap with the given RangeBeginning and RangeEnding times
3. Associate the corresponding granule pass or tile with the given RangeBeginning and RangeEnding times for cataloging and file naming.

External Interface to CNES, GDS, and PO.DAAC

GDS and CNES push the incoming data files to SDS. PCM automatically monitors, detects, and ingests the defined data files at the inbound staging areas. The data files are checked for integrity before they’re ingested into PCM.

...

1. Products are placed in the archive staging location by CNES.
2. PCM generates a CNM-S message which indicates the location of the product. This message is sent to DAAC using AWS Kinesis. 
3. DAAC ingests the product and generates a CNM-R message, indicating success or failure of ingestion. This message is sent to PCM using an SNS topic which is connected to a AWS Lambda.
4. PCM processes and stores CNM-R messages. These messages are cataloged and later queried for activities such as bulk reprociessing, where a large number of previously ingested products must be gathered from the DAAC. 

Image Added

For products generated by the PCM, messages are delivered to both the DAAC and CNES using the following process:

1. PCM generates products
2. PCM creates and copies tar’d SDP datasets to the PCM Archive Staging bucket outgoing directory.
3. PCM creates CNM-S message for product
   1. PCM immediately creates message and sends to PO.DAAC
      1. PO.DAAC ingests tar file
      2. PO.DAAC returns a CNM-R msg of successful/failed ingestion to PCM SDS via an AWS message service
         1. For success msgs, PCM updates catalog
         2. For failed msgs, PCM SDS troubleshoots
      3. Every X minutes, PCM creates a single CNM-S msg with all the tar files placed in the Archive Bucket outgoing directory and places the CNM-S msg in the PCM Msg Bucket CNM-S directory
   2. CNES polls the PCM Msg Bucket CNM-S directory every Y minutes and pulls the msg(s).
      1. CNES deletes the message
   3. CNES pulls the tar files identified in a msg to CNES
   4. CNES creates one or two CNM-R messages with the success and/or failure of the tar copying activity and places the msg into the PCM Msg Bucket CNM-R directory.
   5. CNES notifies distribution center of product(s)

Metrics & Data Accounting

PCM manages an ElasticSearch database for tracking input products, intermediate products, and all reprocessing campaigns. Metrics collected by the PCM system include:

...

• Data Accountability
• Data Product Latency
• SDS system statistics
• Product delivery to PO.DAAC & CNES

Customized Operator Interfaces

The inherited PCM core capability provides helpful and easy-to-view web interfaces including the following:

...

Multiple concurrently operational pipelines for specific processing scenarios including Forward Processing (Persistent), Bulk Processing (Pre-defined input), and ADT Processing (Temporary). Pipelines are deployed in separate AWS accounts. This allows for isolation of permissions, costing, AWS account limits. Certain resources, such as S3 buckets, are shared between accounts. Permissions for read/write access are specific to account and are configureable. PCM's internal deployment and development process is described in the following diagram:

This diagram shows the interactions developers have with with the PCM DEV cluster and how that deployed system flows to I&T and OPS deployments. In this diagram PCM is using Terraform as the PCM Cluster Launcher. PCM's process for any new, operational deployment, is as follows:

1. System engineer notifies SA team of plan to deploy PCM. SA team provides needed cloud credentials to deploy PCM configuration.
2. Deployment engineer configures deployment by running PCM deployment configuration tool. Tool generates PCM deployment config. 
3. Can optionally re-use a pre-generated configuration.
4. Terraform runs, deploying AWS infrastructure and provisioning PCM components within minutes. Pre-build AMIs are used for EC2 instantiation. 
5. PCM pulls docker images for PGEs
6. Terraform completes, providing operator with URLs to operator web interfaces to verify deployment.

When deployed, PCM supports using Amazon EC2 Spot Instance workers:

...