- Created by Kate Sammons, last modified by Topher Allen on Aug 06, 2020
You are viewing an old version of this page. View the current version.
Compare with Current View Page History
« Previous Version 7 Current »
Page Navigation: |
---|
Commonly Used Acronyms
Term | Description |
---|---|
PGE | Product Generating Executor - a set of executable binaries and/or scripts that generate an output file or files of interest |
non-PGE | a set of executable binaries and/or scripts that perform some action of which their output files are inconsequential |
AOI | Area Of Interest |
ARIA | Advanced Rapid Imaging & Analysis |
ES | Elasticsearch |
EW | Extra-Wide Swath |
GRFN | Getting Ready For NISAR |
GUNW | Geo-Unwrap |
GRQ | Geo Region Query, the product catalog holding metadata about all of our products |
HySDS | Hybrid-cloud Science Data System |
IW | Interferometric Wide Swath |
NISAR | NASA-ISRO SAR |
OPS | Operations |
PAGER | Prompt Assessment of Global Earthquakes for Response |
PDL | Product Distribution Layer |
PGE | Product Generation Executive |
PREORB | Precision Orbit |
SAR | Synthetic Aperture Radar |
SLC | Single-Look Complex |
TLA | Three-Letter Acronym |
HySDS Components
Term | Description |
---|---|
Mozart | the HySDS component that orchestrates jobs, job queues, and other resources |
Metrics | machine that acts are a repository of telemetry for all the HySDS machines, HySDS core has a standard set of things sent to metrics, adaptations can capture additional information |
GRQ | the HySDS component that provides the dataset catalog, APIs manipulating them, and various search interfaces |
Verdi | Runs jobs; may be referred to as `worker node` or `worker instance` |
Factotum | A specialized `verdi` instance for processing rule triggers and on-demand requests (small jobs); may also be used for one-off tasks such as connection-throttled data downloads or data source API querying |
CI | Provides continuous integration of versioned PGE/non-PGE containers and automates cluster integration |
Celery | Celery is an open source asynchronous task queue or job queue which is based on distributed message passing |
Tosca | Search for products, operate on products, either get rid of them or input to a pipeline, or add a trigger rule to automate actions based on incoming products, also the web interface for GRQ |
HySDS Concepts
Term | Description |
---|---|
Job | A versioned PGE/non-PGE or instance thereof that is wrapped and encapsulated in a docker container; within the container, PGEs are wrapped to generate datasets |
Job-spec | HySDS job specification; usually refers to the JSON file that contains it `job-spec.json` |
Hysds-io | HySDS I/O specification; usually refers to the JSON file that contains it, `hysds-io.json` |
Worker | A celery worker that executes jobs |
Worker node | A compute instance that runs one or many workers; synonymous with `verdi` node or `verdi` instance |
Dataset | A single, logical product type or instance thereof that encapsulates all files of interest generated by a PGE into a directory along with inherent HySDS-specific JSON files |
Wrapper | An executable binary or script that executes a PGE to generate an output file or files of interest, forms a dataset, and may execute other non-PGE tasks |
Cluster | an instantiation of a group of HySDS component instances composed of (at a minimum) `mozart`, `metrics`, `grq`, `factotum`, and `ci` instances |
Other Terms
Term | Description |
---|---|
Pleiades | Super computer at Ames, used by a variety of projects at NASA/JPL. Being integrated with HySDS Core |
LDAP group | JPL specific way of managing user accounts. Used to manage people’s access to HySDS |
PCM | HySDS consists of the Process Control Management (PCM) services which provide data discovery, workflow orchestration, metrics collection, a special worker node for system and light weight jobs, and continuous integration |
Terraform Scripts | Related to deployment. Makes it easier to stand up the HySDS cluster and configure it |
Docker | Lightweight virtual machine that you can package up your software |
S3 | Simple storage solution. AWS specific, used primarily to store files (input/output products, PGE code, snapshots. Long term storage for data products |
Osaka | Abstraction layer that deals with data set transfer between various end points, operates at a data set level |
AMI | Amazon Machine Image. Used to create virtual machine within EC2. It contains a snapshot, launch permissions, etc. HySDS uses 4 AMIs that correspond to the instances: GRQ, Factotum, Metrics, Mozart |
Elasticsearch | All relevant metadata associated with jobs is stored in Elasticsearch, integrated into HySDS Core services. |
WebDav | Allows you to interact with a directory structure as a we interface and access each worker, a way to expose the working directory on workers via web interface. |
📖 Related Articles: |
---|
|
Have Questions? Ask a HySDS Developer: |
Anyone can join our public Slack channel to learn more about HySDS. JPL employees can join #HySDS-Community
|
JPLers can also ask HySDS questions at Stack Overflow Enterprise
|
🚀 Page Information: |
---|
Was this page useful? |
Contribution History:
|
Find an Error? Is this document outdated or inaccurate? Please contact the assigned Subject Matter Expert: |
- No labels