Index

2:28pm: ASG at 643 dumby-workers on c3.large/us-west-2 hysds-v2-dev-mozart on m4.4xlarge/us-west-2 40% cpu 8.0MB/s network in 5.0MB/s network out hysds-v2-dev-grq on m4.4xlarge/us-west-2 15% cpu 1.1MB/s network in 1.9MB/s network out hysds-v2-dev-metrics

See https://wiki.jpl.nasa.gov/display/hysds/Death+Valley+Torture+Tests https://wiki.jpl.nasa.gov/display/hysds/Death+Valley+Torture+Tests

(The following was documented on https://github.jpl.nasa.gov/hysds-org/general/issues/456 https://github.jpl.nasa.gov/hysds-org/general/issues/456) Use spot fleet instead of Auto Scaling to also test harikiri at scale. Depends on “verdi event stream on fi

bulk reprocessing prep for KACST (and GRFN 50Gbps metrics): take-2 in trinity mode Goal is to load test at NISAR-scales Data Rates and Volumes Forward “keep-up” processing: 1X at 86TB per day Bulk processing: 4X at 344TB per day Concurrent load: 5X at 430

Date at 12pm Pacific Time WebEx info: https://jpl.webex.com/jpl/j.php?MTID=m47a5e7153b3354edcf4422d50aee463f https://jpl.webex.com/jpl/j.php?MTID=m47a5e7153b3354edcf4422d50aee463f Call in: +1-510-210-8882 USA Toll (Access code: 906 603 518) Collaborative

Date 10:00 AM PDT https://jpl.webex.com/jpl/j.php?MTID=m95f0389340ba0619ee3afca2b99e18de https://jpl.webex.com/jpl/j.php?MTID=m95f0389340ba0619ee3afca2b99e18de Meeting number (access code): 904 013 480 Join by phone +1-510-210-8882 USA Toll Participants L

Date & Location Thursday 2019/05/16 @ 9 AM PST Location: 158-210A and WebEx (https://jpl.webex.com/jpl/j.php?MTID=m8ba5f1719bfbed57ce25de777072ad4b https://jpl.webex.com/jpl/j.php?MTID=m8ba5f1719bfbed57ce25de777072ad4b) Participants Frank Greguska Sujen S

ADE Algorithm Development Environment ADT Algorithm Development Team ARD Analysis Ready Data AODS Analysis Optimized Data Services AWS Amazon Web Services GCP Google Cloud Services HEC High End Computing HPC High Performance Computing HySDS Hybri

Chimera is fully customizable. It depends on some configuration files which are dependent on the project. Page Navigation:

You can read about the file: Chimera Config File Decide on adaptation script names for the 3 Chimera steps. e.g. In NISAR, under the adaptation directory, we created the following: precondition_functions.py - The preconditions adaptation script containing

This is a work in progress and is incomplete ssh into factotum and edit /data/export/home/hysdsops/verdi/etc/supervisord.conf Add the following specified to the desired parameters (note some parameters may be different and the parameters are <queue name>

Overview With the resolution of , this added the capability of HySDS Core to optionally be able to support the use of Podman instead of Docker, depending on project needs. The implementation was largely influenced by the analysis that was done on the use

Page Navigation: Confidence Level High This article been formally reviewed and is signed off on by a relevant subject matter expert. Components Elastic Compute Cloud (EC2): Amazon’s web-interfaced cloud computing service. Resizable compute capacity. All

Currently, HySDS uses docker as the default container engine for running the verdi job worker container (verdi) as well as the job containers themselves. However, as of RedHat Enterprise Linux 8 and its various open-source community variants (RockyLinux,

Ticket: https://hysds-core.atlassian.net/browse/HC-259 https://hysds-core.atlassian.net/browse/HC-259 Background information required to understanding the thought process and planning behind integrating A&A to HySDS OpenID Connect 1.0 OpenID Connect https

Pages in this section:

Created by Hook Hua https://wiki.jpl.nasa.gov/display/~hookh on Apr 11, 2016 https://wiki.jpl.nasa.gov/pages/viewpreviousversions.action?pageId=86540457 Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or

Page Navigation: Confidence Level Moderate This article includes input from several JPLers. Multiple subject matter experts can indicate that a page is more frequently reviewed and updated. Introduction This page outlines some of the key concepts of HySD

Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. Caution: If you're changing an existing visualization, you are modifying every i

Navigation:

Navigation: This is a workflow concept implemented using SciFLo. It is a basic skeleton for creating a workflows using SciFlo and generalized to be easily adapted by any project. SciFlo is the workflow infrastructure integrated with the HySDS framework. T

Confidence Level Moderate This article includes input from several JPLers. Multiple subject matter experts can indicate that a page is more frequently reviewed and updated. This file is intended to tell Chimera which module to import and the adaptation c

Confidence Level Moderate This article includes input from several JPLers. Multiple subject matter experts can indicate that a page is more frequently reviewed and updated. This file is collection of key value pairs. It contains useful information needed

Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. Clone dashboard Click "Clone" Name cloned dashboard Click "Confirm Clone" Screen

Created by Lan Dang https://wiki.jpl.nasa.gov/display/~ldang on Aug 02, 2016 https://wiki.jpl.nasa.gov/pages/viewpreviousversions.action?pageId=99682161 Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or

Gerald Manipon edited this page on Mar 21 · 36 revisions https://github.com/hysds/hysds-framework/wiki/Cluster-Setup/_history Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that t

Info about the current organizations using and contributing to HySDS. Also a place for info about getting permissions, creating user accounts, etc, needed to get involved.

Slack: The Slack page should exist for user/collaborator discussion, NOT issue tracking. Slack is not forever nor indexed and we should avoid simple repeat questions and focus on actual chat and discussion. We can have separate channels for each repo as w

HySDS Core Hysds https://github.com/hysds/hysds Mozart https://hysds-core.atlassian.net/wiki/spaces/HYS/pages/199885677/Mozart Verdi https://hysds-core.atlassian.net/wiki/spaces/HYS/pages/199820104/Verdi Tosca SciFlo Osaka GRQ2 https://hysds-core.atlassia

Containerization is defined as a form of operating system virtualization, through which applications are run in isolated user spaces called containers, all using the same shared operating system (OS). A container is essentially a fully packaged and portab

Discussion points: Check Jira if desired functionality already exists or else create a new feature request. Labelling issues Develop and test the new functionality on their cluster and then submit a PR. Voting on change / accepting pull request. How to ha

Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. Steps Fork the target repository Clone repo git clone “insert link” Create a new

Confidence Level Moderate This article includes input from several JPLers. Multiple subject matter experts can indicate that a page is more frequently reviewed and updated. Instructions This document outlines how to create tabular search results in Elast

Confidence Level Moderate This article includes input from several JPLers. Multiple subject matter experts can indicate that a page is more frequently reviewed and updated. This guide describes how to create an auto-scaling fleet queue within HySDS. Log

torresal edited this page on May 24, 2018 · 11 revisions https://github.com/hysds/hysds-framework/wiki/Create-AWS-Autoscaling-Group-for-Verdi/_history Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or c

Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. Step #1 Click "Create New Dashboard" Screen Shot 2021-05-27 at 7.50.22 PM.png St

WHAT IS STAC? The SpatioTemporal Asset Catalog (STAC) specification provides a common language to describe a range of geospatial information. A 'spatiotemporal asset' is a file that represents info about the earth captured in a certain space & time. It pr

Page Navigation: Confidence Level High This article been formally reviewed and is signed off on by a relevant subject matter expert. Create a Trigger Rule This guide demonstrates how to create a trigger rule that notifies a user when a job is completed.

Overview With HySDS Core release version 5, it added the capability to rollover old jobs from Mozart’s Elasticsearch. The associated Jira ticket can be found here: (legacy ticket) Elasticsearch’s Index Lifecycle Manager (ILM) policy is used to determine w

More info coming soon! In the meantime, feel free to ask questions in Stack Overflow https://stackoverflow.jpl.nasa.gov/ or Slack. Also see: GRQ https://hysds-core.atlassian.net/wiki/spaces/HYS/pages/199885383/GRQ

Add your comments directly to the page. Include links to any relevant research, data, or feedback. Summarize this decision in the table below. Type /date to quickly add the due date and @mention the driver, approver, contributors, and informed to keep eve

Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. Delete Panel Click on the icon, then click "Delete from Dashboard." Note: This doesn't delete the

Page Navigation: Confidence Level Moderate This article includes input from several JPLers. Multiple subject matter experts can indicate that a page is more frequently reviewed and updated. Introduction Use this guide to import PGE’s from an existing cod

Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. This document contains the recommendations for deploying, developing, and runnin

Components: hysds https://github.com/hysds/hysds sciflo https://github.com/hysds/sciflo hysds_commons https://github.com/hysds/hysds_commons osaka https://github.com/hysds/osaka prov_es https://github.com/hysds/prov_es mozart https://github.com/hysds/moza

The steps to implement the adaptation is as follows: Create a directory in your project’s repo to host the Chimera adaptation code e.g. nisar_chimera Create a config directory under the adaptation directory created above. Add/ update required configuratio

Navigation:

The HySDS Directory is comprised of Community, Groups and Users. Users are allowed to submit modifications to many pages on the Wiki but some sections can only be edited by certain groups. Our community page links to other platforms such as Slack and Goog

Overview This article describes the various types of documentation included in the HySDS wiki, and guidelines for creating additional documents following the best practices outlined below. Page Navigation: Who is Your Intended Audience? First ask how know

This page will help you create and publish new documentation using HySDS wiki templates and the style guide. Page Navigation: Confidence Level High This article been formally reviewed and is signed off on by a relevant subject matter expert. Recommendat

NOTE: Topics in this site may only be modified by members of the HySDS Documentation Group. Use these resources for reading and printing only. For installation documentation, please refer to the Installation Guide. For reference documentation on job-spec

Navigation:

v3.0.0-rc.4: HySDS framework version 3.0.0-rc.4 (Greta Thunberg) https://github.com/hysds/hysds-framework/releases/tag/v3.0.0-rc.4 https://avatars0.githubusercontent.com/u/387300?s=40&v=4 https://github.com/pymonger pymonger https://github.com/pymonger re

v2.1.0-rc.3: HySDS framework version 2.1.0-rc.3 (Ryloth) https://github.com/hysds/hysds-framework/releases/tag/v2.1.0-rc.3 https://avatars0.githubusercontent.com/u/387300?s=40&v=4 https://github.com/pymonger pymonger https://github.com/pymonger released t

Version: 1.7 → 7.1 Relevant Github Repos/Branches: Removal of Tosca/Figaro user interfaces in favor of hysds_ui https://github.com/hysds/hysds_ui Big Changes PLEASE LOOK AT AND USE THE NEW ELASTICSEARCH UTILITY CLASS: (SOURCE CODE) https://github.com/hysd

Query Changes Changes to Encoding Plus Signs in URLs https://www.elastic.co/guide/en/elasticsearch/reference/7.x/breaking-changes-7.4.html#_changes_to_encoding_plus_signs_in_urls (7.4) Starting in version 7.4, a + in a URL will be encoded as %2B by all RE

Useful links: https://www.elastic.co/blog/whats-new-elasticsearch-7-10-0-searchable-snapshots-store-more-for-less https://www.elastic.co/blog/whats-new-elasticsearch-7-10-0-searchable-snapshots-store-more-for-less https://www.elastic.co/guide/en/elasticse

Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. Export & Re-import Dashboard Click on the settings tab at the very bottom of the left sidebar To

Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. Export Data as a CSV Click on the three dots in the visualization you want to export, then click

Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. Faceting on Panels You can hover over data to see them individually and click on the field to hav

Page Navigation: Confidence Level Moderate This article includes input from several JPLers. Multiple subject matter experts can indicate that a page is more frequently reviewed and updated. HySDS Components: Factotum Topics Helper Worker (factotum) As st

Page Navigation: Who is HySDS for? What kind of data does it process? Where do you get the raw data before it's processed? Is it something I can access too? How? How much is running HySDS on Amazon going to cost me? How much time will it take for me to le

Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. More Information Coming Soon! Related Articles: Have Questions? Ask a HySDS Deve

Navigation:

Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. Operators can identify failed jobs using any precondition failure in the Resourc

Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. Filter Time Period Filter on certain time periods to see data for the specified period. Click on

Navigation:

Navigation:

Page Navigation: Confidence Level Moderate This article includes input from several JPLers. Multiple subject matter experts can indicate that a page is more frequently reviewed and updated. Attached is a reference generic figaro trigger rules json (mainl

Page Navigation: Commonly Used Acronyms Term Description PGE Product Generating Executor - a set of executable binaries and/or scripts that generate an output file or files of interest non-PGE a set of executable binaries and/or scripts that perform some

What is HySDS? The Hybrid Cloud Science Data System (HySDS) is a science data system framework that can leverage a heteregeneous set of worker nodes from private and public clouds as well as virtual and bare-metal machines to perform every aspect of the t

Groups can be used to define access control over in the wiki A Confluence Administrator will need to define these custom groups. The default groups that are already created are administrators, confluence-users and site-admins. Please edit the following li

HySDS Tools: GRQ Topics See also: Dataset Catalog (GRQ) https://hysds-core.atlassian.net/wiki/spaces/HYS/pages/200507537 Navigation:

Navigation:

torresal edited this page on May 24, 2018 · 8 revisions https://github.com/hysds/hysds-framework/wiki/Hello-Dataset/_history Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that th

Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. Dataset Specification: This page details how to author a new dataset type. In or

Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. Introduction In case of catastrophic failure, you would be more likely to backup

Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. Jira ticket:(see https://hysds-core.atlassian.net/browse/HC-26 https://hysds-cor

Page Navigation: Confidence Level Moderate This article includes input from several JPLers. Multiple subject matter experts can indicate that a page is more frequently reviewed and updated. TODO: @Xing, Marjorie Lucas @alphan, Alexander Torres Developer

Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. Jira ticket: (see https://hysds-core.atlassian.net/browse/HC-137 https://hysds-c

Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. The leaflet map allows you to either draw a polygon or a bounding box to constrain your results g

source ~/verdi/bin/activate ~/verdi/ops/hysds/scripts/ingest_dataset.py <product_dir> datasets.json Related Articles: Have Questions? Ask a HySDS Developer: Anyone can join our public Slack channel https://hysds.slack.com/ to learn more about HySDS. JPL e

Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. Faceting On & Purging Similar Failed Jobs: Navigate to the Resource Manager (Fig

Confidence Level High This article been formally reviewed and is signed off on by a relevant subject matter expert. This page outlines how to retry a job in Figaro once its been submitted. Instructions Create a step-by-step guide: Navigate to the Figaro

Page Navigation: Confidence Level High This article been formally reviewed and is signed off on by a relevant subject matter expert. This page outlines how to revoke a job in Figaro once its been submitted. Instructions Create a step-by-step guide: Navi

Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. TODO: Alexander Torres (see https://hysds-core.atlassian.net/browse/HC-95 https:

Originally created by Lan B Dang, last modified on Jul 25, 2017 Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. Intro The X-Fiel

How to Author a New Dataset Type How to run a stand alone MATLAB application locally and in a HySDS cluster How to Create-AWS-Autoscaling-Group-for-Verdi https://github.com/hysds/hysds-framework/wiki/Create-AWS-Autoscaling-Group-for-Verdi How to Create AW

Created by Michael D Starch https://wiki.jpl.nasa.gov/display/~mstarch, last modified on Feb 13, 2017 https://wiki.jpl.nasa.gov/pages/diffpagesbyversion.action?pageId=128433468&selectedPageVersions=1&selectedPageVersions=2 Page Navigation: Confidence Leve

Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. Introduction HySDS has a number of APIs that are designed to provide a way for e

Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. Highlights Scale-up is done by an auto-scaling up metric alarm that checks the q

HySDS UI has a minimal dashboard in Kibana to display current status for its services (supervisor / systemd) Kibana is running on https://<MOZART_IP_ADDRESS>/metrics/app/home Accessing the HySDS Cluster Health page Fig 1. Link to Kibana in HySDS UI link.p

More info coming soon! In the meantime, feel free to ask questions in Stack Overflow https://stackoverflow.jpl.nasa.gov/ or Slack.

Navigation:

Page Navigation: Confidence Level High This article been formally reviewed and is signed off on by a relevant subject matter expert. Agenda develop -> develop-es1 run end-to-end smoke test to test develop-es1 create release v3.0.1 from develop-es1 run e

Page Navigation: Facet Search Available for metrics, resources Metrics Available for jobs, infrastructure Trigger Rules Enact automated and customized processing and analysis Jobs Resiliency Generic trigger rules enable handling of common failed jobs Supp

Page Navigation: Confidence Level High This article been formally reviewed and is signed off on by a relevant subject matter expert. This page introduces the main GUI’s used throughout HySDS. A brief description of their function and instructions on how

Kubernetes https://kubernetes.io/ is an open-source system for automating deployment, scaling, and management of containerized applications. K8 pods run an instance of a docker image, similar to a docker container. We can run all of our services in HySDS

This site is a collection of definitions and descriptions of HySDS concepts, tools and procedures. see also: Beginner's Guide to HySDS FAQs Glossary Container Technology Jobs- Jobs are subclasses of Tasks that add notions of running a sub-process executab

Confidence Level High This article been formally reviewed and is signed off on by a relevant subject matter expert. This page describes HySDS - including its six main components and their respective purposes. For a more detailed look at each of these co

Confidence Level High This article been formally reviewed and is signed off on by a relevant subject matter expert. This page contains a more detailed view of the resources within each of the HySDS components. Below is a diagram of each of the six compo

Page Navigation: Confidence Level Moderate This article includes input from several JPLers. Multiple subject matter experts can indicate that a page is more frequently reviewed and updated. This article outlines some of the most common log files operator

You can read the introduction here: Input Preprocessor Create the script mentioned in the chimera_config.yaml file under the preprocessor.module_path. In the script, declare the subclass mentioned in the chimera_config.yaml file under the preprocessor.cla

You can go through the introduction here: PGE Execution Create the script mentioned in the chimera_config.yaml file under the job_submitter.module_path. In the script, declare the subclass mentioned in the chimera_config.yaml file under the job_submitter.

You can go through the introduction here: Post Processor Create the script mentioned in the chimera_config.yaml file under the postprocessor.module_path. In the script, declare the subclass mentioned in the chimera_config.yaml file under the postprocessor

Confidence Level Moderate This article includes input from several JPLers. Multiple subject matter experts can indicate that a page is more frequently reviewed and updated. This is the first step of Chimera called Input Preprocessor (IPP) The Input prepr

Hook Hua edited this page on Jun 12, 2018 Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. Installation For definitions of termin

For definitions of terminology used, please refer to our terminology reference https://github.com/hysds/hysds-framework/wiki/Docs#general-hysds-terminology. For trouble shooting this guide please refer to our Notes on Installation https://hysds-core.atlas

Navigation:

Confidence Level Moderate This article includes input from several JPLers. Multiple subject matter experts can indicate that a page is more frequently reviewed and updated. The HySDS Resource Manager (called “Figaro”) allows operators to monitor the stat

Confidence Level Moderate This article includes input from several JPLers. Multiple subject matter experts can indicate that a page is more frequently reviewed and updated. This page introduces the HySDS Core graphical user interface (GUI), nicknamed “To

Track long-running issues and various mitigation steps with the goal of finding a long-term solution. Issues and Mitigation Navigation:

Michael Cayanan edited this page on Jul 11 · 30 revisions https://github.com/hysds/hysds-framework/wiki/Job-and-HySDS-IO-Specifications/_history Page Navigation Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or complete

Confidence Level High This article been formally reviewed and is signed off on by a relevant subject matter expert. This article outlines the nominal data flow of a job though HySDS. job status_updated6-8.jpg Submit a Job A job is submitted via On-Deman

Singularity port of Verdi to run on NASA Pleiades HEC TBD: Lei.Pan@jpl.nasa.gov

Navigation:

Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. Kibana Query Language (KQL) The Kibana Query Language (KQL) is a simple syntax for filtering Elas

Confidence Level High This article been formally reviewed and is signed off on by a relevant subject matter expert. HySDS offers several job management options. After faceting on the desired jobs in the Resource Management Interface (called Figaro), the

Load Tests Navigation:

Navigation:

Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. See background : NASA HECC Pleiades Setting up Tunnel on Pleiades head node back

Page Navigation: Confidence Level High This article been formally reviewed and is signed off on by a relevant subject matter expert. This article demonstrates how to perform various commonly used tasks with Amazon Web Services (AWS) which are applicable

Incomplete tasks from meetings All meeting notes

Navigation:

HySDS Components: Mozart Topics Navigation:

Originally created by Hook Hua, last modified on Mar 16, 2017 Page Navigation: Confidence Level Moderate This article includes input from several JPLers. Multiple subject matter experts can indicate that a page is more frequently reviewed and updated. RE

Hook Hua edited this page on Aug 28, 2018 · 7 revisions https://github.com/hysds/hysds-framework/wiki/Mozart-Watchdog-Notifications/_history Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completenes

Created by Hook Hua https://wiki.jpl.nasa.gov/display/~hookh, last modified by Lan Dang https://wiki.jpl.nasa.gov/display/~ldang on May 05, 2017 https://wiki.jpl.nasa.gov/pages/diffpagesbyversion.action?pageId=86540519&selectedPageVersions=18&selectedPage

Page Navigation: Confidence Level High This article been formally reviewed and is signed off on by a relevant subject matter expert. Intro To leverage the computing power of pleiades, we port verdi to run bare-metal on pleiades and have it communicate w

Confidence Level Moderate This article includes input from several JPLers. Multiple subject matter experts can indicate that a page is more frequently reviewed and updated. In HySDS, the Elasticsearch (ES) database can be accessed via the “es:head” link

Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. I followed the installation instructions found here https://hysds-core.atlassian.net/wiki/spaces/

Sources: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless.html https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless.html Cost OpenSearch serverless needs a "server" running called a OCU (OpenSearch

Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. In OpenStack Console Create a volume 200-300GB In OpenStack, go to Volumes Volum

Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. note This page should record general operational issues we have seen in operatin

Operations Navigation:

hysds logo.png The HySDS Operations Manual contained in this section provides documentation to the most common features and functionality in HySDS core. The intent of these documents is to provide new HySDS Operations Engineer with a point of reference fo

Mozart Instance r5.2xlarge: 64 GB RAM, 8 vCPUs, 10 Gb network Kernel parameters: sysctl /usr/lib/sysctl.d/mozart.conf: # Redis and Elasticsearch #recycle sockets quickly net.ipv4.tcp_tw_recycle=1 # backlog setting net.ipv4.tcp_max_syn_backlog=102400 #Max

Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. Premise Some sling jobs get terminated (for a variety of reasons in AWS such as

More info coming soon! In the meantime, feel free to ask questions in Stack Overflow https://stackoverflow.jpl.nasa.gov/ or Slack.

Your feedback is valuable. Please complete this brief survey to help us understand how we can improve our documentation:

Thank you for your feedback. Your input helps us improve our wiki documentation .

Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. Excerpts below from: HySDS APIs Pele Links and Examples Code can be found in: ht

Confidence Level Moderate This article includes input from several JPLers. Multiple subject matter experts can indicate that a page is more frequently reviewed and updated. This YAML file is the key that drives what preconditions are evaluated, what kind

Confidence Level Moderate This article includes input from several JPLers. Multiple subject matter experts can indicate that a page is more frequently reviewed and updated. This is the second step of Chimera. It takes the configuration generated by IPP a

More info coming soon! In the meantime, feel free to ask questions in Stack Overflow https://stackoverflow.jpl.nasa.gov/ or Slack.

Please review the list of placeholder articles, and leave a comment at the bottom of this page which you would find most valuable. We have also already identified priority articles (highlighted in pink) and potential individuals to be responsible for som

Decision Blueprint Road Map

Created by Hook Hua https://wiki.jpl.nasa.gov/display/~hookh, last modified on Apr 01, 2016 https://wiki.jpl.nasa.gov/pages/diffpagesbyversion.action?pageId=86540471&selectedPageVersions=3&selectedPageVersions=4 HySDS can run in multiple different platfor

podman is: … a daemon-less container engine for developing, managing, and running OCI Containers on your Linux System For the most part podman behaves very similarly to docker, with much of the commands being identical, ie: podman images - lists all image

Confidence Level Moderate This article includes input from several JPLers. Multiple subject matter experts can indicate that a page is more frequently reviewed and updated. This is an optional step. The main task of the post processor is to run any post

Gerald Manipon edited this page on May 16 · 7 revisions https://github.com/hysds/hysds-framework/wiki/Preprocessor-and-Postprocessor-Functions/_history Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Che

Page Navigation: Confidence Level Moderate This article includes input from several JPLers. Multiple subject matter experts can indicate that a page is more frequently reviewed and updated. This article provides a brief overview of the .JSON log files fo

Confidence Level Moderate This article includes input from several JPLers. Multiple subject matter experts can indicate that a page is more frequently reviewed and updated. This file should contain the field names found in the product’s metadata (in the

EXPLORE HOW HySDS IS BEING USED Currently HySDS is integrated into several projects at JPL and elsewhere. It was built at JPL for the purpose for supporting research in areas of radar and satellite data processing, weather and climate modeling and data as

Created by Hook Hua https://wiki.jpl.nasa.gov/display/~hookh, last modified on Oct 01, 2014 https://wiki.jpl.nasa.gov/pages/diffpagesbyversion.action?pageId=86540483&selectedPageVersions=15&selectedPageVersions=16 Page Navigation: Confidence Level TBD Th

Relevant Guide(s) Create your own Python package and publish it into PyPI https://towardsdatascience.com/create-your-own-python-package-and-publish-it-into-pypi-9306a29bc116 https://packaging.python.org/en/latest/ https://packaging.python.org/en/latest/ h

Gerald Manipon edited this page on Sep 13, 2018 · 3 revisions https://github.com/hysds/hysds-framework/wiki/Puppet-Automation/_history Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Che

Page Navigation: Confidence Level High This article been formally reviewed and is signed off on by a relevant subject matter expert. We are working on porting our python 2 PGE's to python 3. Individual PGE's will be tested and validated against equivale

Context: I know that HySDS persists most of its state in Elasticsearch. How can I regularly backup my Elasticsearch indices? Answer: In case of catastrophic failure, you would be more likely to backup GRQ and Metrics Elasticsearch indices as your products

Context: Via Alex Dunn regarding a HySDS pge he maintains: Due to how the PGE runs, there will be an overlap in the products it creates with products that are already in the system. In this case, the PGE will error out with a no-clobber error instead of c

Answer: In Figaro, there is a category of job state called "Offline." What is the operator supposed to do with these offlien jobs?perator supposed to do with Offline jobs?

Context: I am working with a HySDS system. I am trying to remove a PGE that was installed on this HySDS system. I tried using sds ci remove https://github.com/aria-jpl/create_aoi_track But it failed with an error involving jenkins. How do I fix it? Answer

Context: As a developer, I need to update datasets.json on factotum, so it knows about my new product type. But my changes keep getting overwritten? How do I properly update datasets.json? Answer: It's my understanding that you just add your entry to ~/.s

Context: To avoid no-clobber job errors https://stackoverflow.jpl.nasa.gov/questions/1533 when ingesting overlapping datasets, it has been suggested that json-only datasets can be set to avoid publishing anything to S3 and instead rely on (overwritable) E

Page Navigation: Confidence Level High This article been formally reviewed and is signed off on by a relevant subject matter expert. RabbitMQ Overview RabbitMQ is used in combination with Celery to queue jobs. Logging into RabbitMQ can give you a broade

Page Navigation: Confidence Level Moderate This article includes input from several JPLers. Multiple subject matter experts can indicate that a page is more frequently reviewed and updated. Intro RabbitMQ has a web UI to see real-time information about q

This template is a guide to help you create your Release pages based on Confluence recommendations from this page: https://confluence.atlassian.com/doc/permissions-and-restrictions-139557.html https://confluence.atlassian.com/doc/permissions-and-restricti

Gerald Manipon edited this page on Jun 24 · 5 revisions https://github.com/hysds/hysds-framework/wiki/Releases/_history Releases: Page Navigation: Confidence Level Moderate This article includes input from several JPLers. Multiple subject matter experts

Use this procedure to remove actions from the actions list in Tosca. This is helpful for keeping the list short and current. Log into the Mozart instance Get the list of packages ex. sds pkg ls Copy the name of the package you want to remove then run the

Upcoming Plans and Features Being Planned and Discussed Community Planning

Created by Gerald John M Manipon https://wiki.jpl.nasa.gov/display/~gmanipon, last modified on Nov 21, 2016 https://wiki.jpl.nasa.gov/pages/diffpagesbyversion.action?pageId=127176782&selectedPageVersions=2&selectedPageVersions=3 Page Navigation: Confidenc

Navigation: SciFlo is a workflow framework that comes with core HySDS. It allows developers to create workflows by writing up a workflow definition document. SciFlo takes the workflow definition document and runs through all the steps as a workflow job. E

Page Navigation: Confidence Level High This article been formally reviewed and is signed off on by a relevant subject matter expert. As briefly mentioned before SciFlo is a workflow framework. It uses a XML workflow definition document to know the seque

Created by Gerald John M Manipon https://wiki.jpl.nasa.gov/display/~gmanipon on Nov 21, 2016 https://wiki.jpl.nasa.gov/pages/viewpreviousversions.action?pageId=127176877 Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or

Page Navigation: Confidence Level High This article been formally reviewed and is signed off on by a relevant subject matter expert. A Worflow Definition document tells SciFlo: What to run how to run it what the inputs and outputs are The order of the s

Page Navigation: Confidence Level High This article been formally reviewed and is signed off on by a relevant subject matter expert. Overview SDSWatch is designed to gain insight into what compute nodes are doing in real-time, and Kibana dashboard / vis

Page Navigation: Confidence Level High This article been formally reviewed and is signed off on by a relevant subject matter expert. Intro SDSWatch is a logging mechanism targeting the collection of insights and metrics. gbo0RSQFpCwhA-mMiimDSb7SOQOmBXrr

Popular Labels Browse by Topic

Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. Seeing spikes in workflow duration Kibana includes a free Data Visualizer to learn more about you

Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. Seeing the existing data in an index pattern Kibana requires an index pattern to access the Elast

Confidence Level Moderate This article includes input from several JPLers. Multiple subject matter experts can indicate that a page is more frequently reviewed and updated. The settings.yaml file is something that exists under conf directory for missions

Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. There are three ways to share a dashboard. Click on "Share" >> "Permalink" Snapshot: Shares the d

Confidence Level High This article been formally reviewed and is signed off on by a relevant subject matter expert. Shareable URL links of custom faceted search criteria can be created within HySDS Facet Search (Tosca) as well as the HySDS Resource Mana

This is the version 1.0 (2019-05-17) Software Design Document (SDD) for the Hybrid Cloud Science Data System (HySDS). Introduction Purpose This document provides the high level design information to understand the architecture and design of the core HySDS

Question backup ElasticsearchQuestion offline jobsQuestion update datasets.json on factotumQuestion valid non-published datasets.json definitionQuestion enable clobberingQuestion remove PGE from SDS

Question Answer Slack Link Comments 1 I am working with a HySDS system. I am trying to remove a PGE that was installed on this HySDS system. I tried using sds ci remove https://github.com/aria-jpl/create_aoi_track But it failed with an error involving jen

Page Navigation: Confidence Level Moderate This article includes input from several JPLers. Multiple subject matter experts can indicate that a page is more frequently reviewed and updated. The Science Data System (SDS) command line interface provides nu

Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. We will provision the HySDS component images and then use terraform to deploy th

Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. The latest releases are here: https://github.com/hysds/hysds-framework/releases

Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. Prerequisites Install Terraform from https://www.terraform.io/ https://www.terra

Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. When installation of the HySDS framework is complete on your mozart instance (se

Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. *With the OPS_USER and OPS_PASSWORD, you can use the https (e.g. https:// ) prot

Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. Step 1 In this tutorial, we continue where we left off in integrating our first

Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. Prerequisites You should have already created an autoscale AMI. If it is not pro

Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. Setting up an AWS Resources for Staging Area For definitions of terminology used

Navigation:

Page Navigation: Introduction This page provides an overview of the styling for all HySDS public wiki documentation. The goal of this guide is to provide a useful reference while creating new documentation while standardizing the wiki’s content. Voice: To

Confidence Level Moderate This article includes input from several JPLers. Multiple subject matter experts can indicate that a page is more frequently reviewed and updated. HySDS jobs can be submitted for immediate processing using the “On-Demand” button

Introduction Purpose This document provides the high level design information to understand the architecture and design of the core HySDS framework. It is not intended to replace reading the code for details. Background The primary responsibility of a sci

Hook Hua edited this page on Jul 9, 2018 · 5 revisions https://github.com/hysds/hysds-framework/wiki/System-Port-Map/_history Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information i

Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. What are HySDS Tasks? Tasks are Celery-level python functions that are dispatche

Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. Unit Tests Unit tests for individual projects can be run using the standard pyth

Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. Components HySDS integrates together the following 3rd Party services and software: C

Page Navigation: Confidence Level High This article been formally reviewed and is signed off on by a relevant subject matter expert. Status of standard product and urgent response thread-tests pipelines in Mamba cluster. Standard Product Pipeline PGE Re

Priority Deliverable Notes Status Blockers Internal HySDS wiki “how-to” page dispositions Finished dispositions in “how-to” section Forwarded ARIA related docs https://docs.google.com/document/d/1-gIxJCaPITmNRkCL3CmnHOHpzHPGUwsyWWrpwUPmCRo/edit to Marjori

Page Navigation: Confidence Level High This article been formally reviewed and is signed off on by a relevant subject matter expert. Intro This resource management UI, also known as Tosca, is a custom built web application adapted from the FacetView pro

Navigation:

Note: A full breakout of the underlying components are documented here in the SDD. For the purposes of this discussion, we focus on the key and driving components relevant for increasing the scale of total distributed workers. Basic Deployment between Moz

Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. Understanding Current Metrics There are three main dashboards for HySDS: Dashboard Purpose Import

Confidence Level Moderate This article includes input from several JPLers. Multiple subject matter experts can indicate that a page is more frequently reviewed and updated. This guide walks the user through the steps required to deploy a HySDS core relea

Read the Settings File to learn about the purpose of the file. Set the PGE_SIMULATION_MODE to true or false. Set the JOB_TYPES AND JOB_QUEUES for every PGE. The PGE name should match the pge_name defined in the PGE Config File. The job types should match

Gerald Manipon edited this page on Aug 30, 2018 · 6 revisions https://github.com/hysds/hysds-framework/wiki/Upgrade/_history Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that th

What's new in Python 3.9 https://docs.python.org/3/whatsnew/3.9.html Related ticket(s): https://hysds-core.atlassian.net/browse/HC-385 https://hysds-core.atlassian.net/browse/HC-385 https://jira.jpl.nasa.gov/browse/NSDS-1555 https://jira.jpl.nasa.gov/brow

Please add yourself to the list by filling out the form in HySDS Users registration. This will create an account which enables you to edit topics. JPL: Hook Hua Gerald Manipon Lan Dang Mohammed Karim Albert Li Dustin Lo Sujen Shah Michael Cayanan Marjorie

Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. Release Notes for HySDS framework version 3.0.0 (Black Mamba) Github release: ht

Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. Release Notes for HySDS framework version 3.0.0-rc.5 (LeBron James) Github relea

Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. Release Notes for HySDS framework version 3.0.0-rc.6 (Anthony Davis) Github rele

Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. Release Notes for HySDS framework version 3.0.1 (Medical Professionals) Github r

Date Mar 12, 2021 Issues Summary HySDS framework (core) v4.0.0 includes bug fixes and major enhancements. Highlights migrate from using Elasticsearch 1.7 to Elasticsearch 7.1 support using AWS Elasticsearch Service for GRQ backend ES ELK stack 7.1 on metr

Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. Date Aug 05, 2020 Issues Summary HySDS framework (core) v4.0.0-rc.0 includes bug

Date Nov 04, 2020 Issues Summary HySDS framework (core) v4.0.0-rc.3 includes bug fixes and major enhancements. Highlights Major cleanup of deprecated code Updates to improve ES 7.1 performance remove document bloat in dataset ES JSON payload set default n

Date Mar 23, 2022 Issues Summary HySDS framework (core) v4.0.1 includes bug fixes and major enhancements. Highlights Updated to support Python 3.9 Migrate from using Elasticsearch 7.1 to Elasticsearch 7.9 support using AWS Elasticsearch Service for GRQ ba

Date Aug 11, 2022 Issues 3 issues Summary HySDS Framework (core) v4.1.0-beta.0 includes bug fixes and major enhancements. Highlights Added transactional publishing of datasets so that Verdi will ensure that those jobs that produce multiple outputs will al

Date Sep 26, 2022 Issues Summary HySDS Framework (core) v4.1.0-beta.2 includes bug fixes and major enhancements. Highlights Added transactional publishing of datasets so that Verdi will ensure that those jobs that produce multiple outputs will all be succ

Verdi Topics

Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. Intro With recent cybersecurity requirements, Verdi compute nodes are locked dow

Page Navigation: Confidence Level High This article been formally reviewed and is signed off on by a relevant subject matter expert. The AWS Management Console provides centralized access to the Amazon cloud computing services utilized in HySDS, includi

Hook Hua edited this page on Aug 23, 2018 · 1 revision https://github.com/hysds/hysds-framework/wiki/Watchdogs/_history Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this in

HySDS: Hybrid-Cloud Science Data System What is HySDS? The Hybrid Cloud Science Data System, or HySDS (pronounced Hi-S-D-S) is an open source science data system that can be used to process large amounts of science data products. It has been used across s

Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. his serves as a list of pages that need work, each page is labelled according to what stage it is

Contains pages related to development and design of the wiki. Wiki Development Pages:

wiki workflow.png Related Articles: Have Questions? Ask a HySDS Developer: Anyone can join our public Slack channel https://hysds.slack.com/ to learn more about HySDS. JPL employees can join #HySDS-Community https://jpl.slack.com/archives/C99QZ9THS JPLers

This is an informal list of features that would facilitate operations. These will not necessarily be implemented, but used to gauge what and where the pain points are. Feature Description Justification Value Reporting Project Reporter enumeration reports

Gerald Manipon edited this page on May 17, 2018 · 8 revisions https://github.com/hysds/hysds-framework/wiki/Workers/_history Page Navigation: Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that th

Navigation:

You can read about the file here: PGE Config File Start by creating a file named PGE_<name>.yaml. We usually store all PGE configs in a common directory. The location to the PGE config file is then mentioned in the SciFlo Workflow Definition file. The bas