HySDS Ops Doc Development Table-Spring '20

Cells with blue background represent input by ARIA/GRFN.


Purple = in progress or ready for feedback

Red = too mission specific, removed

Gold = Needs clarification

Blue = completed, signed-off

count

priority

topic

Status

wiki link

Jira link

comments

count

priority

topic

Status

wiki link

Jira link

comments

 

I'd like to see the How-To's placeholder on the wiki filled in with:

 

 

 

 

1

1

Jira #593: How to track Jobs (thru the system) --nominal flow

5/12 received tag-up clarification, starting v1

5/14 Reviewed with Marjorie, received more input, revising v2.

v2 updated, new image created

6/9 signed off

 Job Workflow in HySDS

https://jira.jpl.nasa.gov/browse/NSDS-593

 

 nominal flow from task to job to …(some of this is probably generic and some mission specific

mlucas: approved

2

1

JIRA #594: How to manage Jobs  (seeing, stopping, removing, re-submitting, following) (aka “lightweight” jobs”)

5/5 TAG UP CLARIFICATION:

  1. “would like to see more info on tracking the job thru different states in the system”

  2. Related to/duplicate of Jira #593 above?

4/28 Revised v2

6/9 revised, possible updates needed

 Lightweight Jobs in Resource Manager

https://jira.jpl.nasa.gov/browse/NSDS-594

 would like to see more info on tracking the job thru different states in the system

mlucas:

3

1

JIRA #595: How to purge all jobs by a time range (See #601, #597)

Potentially implemented as a how-to page

6/2 tag up: likely fits in a Step-by-step guide. The key is using the ES query for a timerange. Marjorie is going to follow-up with the correct query string

@timestamp: {2020-06-08T00:00:00 TO *} (used for Figaro/Tosca)

5/14 reviewed with Marjorie, she’s going to follow up

6/9 see query string. Implement as a cheatsheet that includes the timestamp query <--

 

https://jira.jpl.nasa.gov/browse/NSDS-595

 

4

1

Jira #598 How to identify gaps in data

6/2 probably too mission specific.

6/2 confirm this is too mission specific with Lela

6/9 Lela agrees.

 

 

 

5

1

JIRA #597: How to facet on all failed jobs with the same failure message, purge them and then re-submit them (Related to #601)

6/9 After tag up with Marjorie, added How to Retry and How to Revoke stub articles

4/16 received v2 feedback

4/30 updating based off feedback

6/9 added two articles -->

6/9 pending comment clean-up, otherwise signed-of

 How to Purge Jobs

 

How to Retry Jobs

 

How to Revoke Jobs

https://jira.jpl.nasa.gov/browse/NSDS-597

 

6

1

How to know the list of all the log files involved with HySDS and the top 3 that operators should become familiar with (**Overlap with ARIA/GRFN request below)

5/5 TAG UP CLARIFICATION:

  1. Explain log files? What should the article scope cover?

6/9 Marjorie will provide me with location of the logs, the types of logs, description of each.

(Note: not necessary to list/describe contents of each log file)

6/11 completed v1 based on log files/descriptions provided by Marjorie

  HySDS Log File Overview

 

 

 metadata is indeed useful to know about but topic was for HySDS “log” files NSDS-796-ljr

7

1

JIRA #599: How to know when operators need to use potentially different guis... like celery, rabbitmq, aws and kibana

{6/2 And Tosca and Figaro}

4/30 Started v1, finished minimally viable v1, ready for feedback

5/5 incorporated initial feedback re “how”, added additional images. No SWOT queue, AWS access. v1.5 in progress

6/4 updated with additional GUIs

6/9 Marjorie will fill in blanks with remaining GUI’s

6/11 updated Tosca and Figaro content.

 HySDS GUI's Overview

 

https://jira.jpl.nasa.gov/browse/NSDS-599

 looking for the “why” operator would need to access different guis, not just the “how”

8

1

JIRA #600: How to tell which products have not been produced yet

5/5 TAG UP CLARIFICATION:

  1. Confirm if too mission specific? → yes.

 

Confirmed in tag up, Too mission specific

 

https://jira.jpl.nasa.gov/browse/NSDS-600

Check w Lela for clarification; this might be mission specific(?)

9

1

JIRA #601: How to filter on failed jobs due to precondition failure

(Note: precondition failure “is when a PGE fails not because of a runtime error but due to missing inputs, corrupt inputs, etc. These show up as failed jobs… Not something easily visible. These aren’t necessarily distinguished from a failed job”)

5/5 pending v2 revisions

 

6/9 Marjorie will get more clarification from HySDS team (Gerald, Namrata)

 Filter on Failed Jobs due to Precondition Failure

https://jira.jpl.nasa.gov/browse/NSDS-601

 LR: added comment on wiki page link

10

1

How to determine latency between receipt of raw telemetry and production of pipeline products

4/30 received feedback on this from Namrata; likely too mission specific

 

 

 LR: sign off

concur too mission specific

11

2

JIRA #603: How to query Elastic search catalog by filename, ingest time  and get a tabular search result

5/5 TAG UP CLARIFICATION:

  1. Tabular search results = ?

  2. “would like the example expanded to include the GRQ es (not just the mozart one)”

 Navigating the Elasticsearch Plug-in head Database

https://jira.jpl.nasa.gov/browse/NSDS-603

would like the example expanded to include the GRQ es (not just the mozart one)

12

2

JIRA #604: (Elasticsearch) Like to know how to see all the different indexes and what all can be queried

5/5 TAG UP CLARIFICATION:

  1. How to get tabular search results:

  2. “add structured query example”

  3. “add search results image“

6/4 reviewed with Marjorie

 

6/9 revised article, linked to stub article outlining tabular search results -->

6/9 TODO: append webex screenshot and add comment with alternate method Done.

 Navigating the Elasticsearch Plug-in head Database

 

Create a Tabular View of ES Documents

https://jira.jpl.nasa.gov/browse/NSDS-604

 add search results image and add structured query example

with how to get the tabular results

13

3

JIRA #605: Start/stop/restart/status/reset and supervisord role up next

4/30 pending tag up w Namrata for 5/1

5/10 v1 done, pending tag-up w Marjorie for revisions

v2 revised, image added

6/9 reviewed, signed-off

 Start, Stop, Restart, Status, Reset, Supervisord

https://jira.jpl.nasa.gov/browse/NSDS-605

 

14

3

JIRA #606: how to find and use any/all PCM config files (settings.yaml)

4/30 received feedback on this from Namrata; likely too mission specific

 

https://jira.jpl.nasa.gov/browse/NSDS-606

 LR: sign off

concur too mission specific

15

 

JIRA #607: Create/edit/delete trigger rules

4/28 Finished v1, ready for feedback

5/5 incorporated remaining feedback. Done..

Create, Edit, & Delete Trigger Rules

https://jira.jpl.nasa.gov/browse/NSDS-607

LR: sign off

 

16

 

JIRA #608: Submit on-demand jobs via Figaro and Tosca

4/14 Finished v1 (Figaro)

4/10 Finished v1 (Tosca)

4/28 Revised, ready for sign-off

Submit an On-Demand Job in Facet Search

Lightweight Jobs in Resource Manager

https://jira.jpl.nasa.gov/browse/NSDS-608

LR: sign off

17

 

**Description of product log file types (e.g. _context.json, _met.json, _dataset.json)

4/14 Finished v1

4/28 Revised, ready for sign-off…

5/5 Done.

https://hysds-core.atlassian.net/wiki/spaces/HYS/pages/389349377

 

LR: sign off

 

18

 

JIRA #610: Diagram of interfaces between HySDS components showing data flow

4/26 Finished L1 + L2 v1

4/28 Ready for feedback

5/5 working on Marjorie’s feedback

5/10 Uploaded v2 L1 & L2 with revisions, feedback received, working on v3

L1 v3 finished; made text searchable per feedback

6/4 Updated L1 and L2, added CI to L2

6/9 signed-off

https://hysds-core.atlassian.net/wiki/spaces/HYS/pages/506232987

 

https://hysds-core.atlassian.net/wiki/spaces/HYS/pages/506233031

https://jira.jpl.nasa.gov/browse/NSDS-610

 

19

 

JIRA #612: Troubleshoot frequently-seen issues

6/2 TAG UP CLARIFICATION: (FYI) the intent is to have a place where people can write up solutions to frequently seen problems in HySDS core.

5/14 tag up with Marjorie, she’s going to follow up with me with specific examples

6/2 Deemed lower priority per Marjorie

6/9 Needs discussion regarding best implementation for HySDS community

 

https://jira.jpl.nasa.gov/browse/NSDS-612

 

20

 

JIRA #613: View AWS running instances, buckets, and ASG’s (auto scaling groups)

4/28 Lan tag-up, working on v1

5/1 Pending AWS access; doesn’t look like I have it.

5/10 Finished v1, Ready for feedback

5/14 Awaiting feedback from Marjorie on v1

6/9 signed-off

https://hysds-core.atlassian.net/wiki/spaces/HYS/pages/520945758

https://jira.jpl.nasa.gov/browse/NSDS-613

 

21

 

JIRA #614: Basic manipulation of AWS instances, buckets, and ASG’s via CLI and GUI

Clarification: how to stop an instance, how to terminate an instance, creating a bucket, also describe how to get an IP of an EC2 instance. For ASG’s: knowing how to change desired and maximum capacity.

5/1 Pending AWS access; doesn’t look like I have it.

6/2 working on v1

6/4 finished v1

6/9 signed-off

https://hysds-core.atlassian.net/wiki/spaces/HYS/pages/575045633

https://jira.jpl.nasa.gov/browse/NSDS-614

 

22

 

Define resource types within job management

5/5 TAG UP CLARIFICATION:

4/28 need clarification (task, job, event, worker, etc)

6/9 incomplete

 

 

 

23

 

JIRA #616: Send accessible links of faceted content

 

4/11 finished v2, done.

https://hysds-core.atlassian.net/wiki/spaces/HYS/pages/395083799

https://jira.jpl.nasa.gov/browse/NSDS-616

LR: sign off

24

 

Incorporate generic Mozart trigger rules into Operations Docs as appropriate. Link to generic rules on public wiki.

surgically delete an entry (exists on old internal documentation) update to public

 

 

6/2 reviewed with Marjorie, request is tentatively satisfied with linked doc →

6/9 signed-off

Create, Edit, & Delete Trigger Rules

 

Lan’s suggestion via Slack

Hierarchy of HySDS Operator Manual (current 6/9/20):

Jira dashboard: https://jira.jpl.nasa.gov/secure/Dashboard.jspa#

 

Related Articles:

Have Questions? Ask a HySDS Developer:

Anyone can join our public Slack channel to learn more about HySDS. JPL employees can join #HySDS-Community

JPLers can also ask HySDS questions at Stack Overflow Enterprise

 

Search HySDS wiki

Page Information:

Was this page useful?

Yes No

Contribution History:

No contributors found for: authors on selected page(s)

Find an Error?

Is this document outdated or inaccurate? Please contact the assigned Subject Matter Expert:

@Topher Allen

 

Note: JPL employees can also get answers to HySDS questions at Stack Overflow Enterprise: