Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

(blue star) Confidence Level Moderate  This article includes input from several JPLers. Multiple subject matter experts can indicate that a page is more frequently reviewed and updated.


This YAML file is the key that drives what preconditions are evaluated, what kind of job is submitted and what post processing steps need to be run.

A PGE config file consists of the following elements:

  • Runconfig template:
    The runconfig section is intended to be used for dynamically generating the runconfig file used when running the NISAR PGEs. Any values set with a "null" should be filled in by the input preprocessor. Otherwise, a precondition evaluation error will be thrown.

  • Precondition functions to run:
    This lists all the precondition evaluation steps that this PGE needs to run prior to running the PGE.

  • additional information for precondition functions (if required)
    For any of the precondition evaluation steps listed in the preconditions area, we specify function arguments.

  • PGE name:
    name of the PGE

  • Primary input to PGE:
    specify primary dataset type that’s input to the PGE

  • PGE Output file types:
    List the extensions that the PGE generates

  • Files to be downloaded by PGE execution job:
    List the groups within the run config that the PGE job will need to localize

  • Mock metadata (if PGE run is being mocked):
    When doing simulated PGE runs, this will mock together a set of metadata for the met.json output.

  • PGE Output file’s base name (if PGE run is being mocked):
    When performing simulated PGE runs, this sets the base name to be used when generating the output products.

Here is an example PGE config:

Code Block
languageyaml
# The runconfig section is intended to be used for dynamically generating the runconfig file used
# when running the NISAR PGEs. Any values set with a "null" should be filled in by the input
# preprocessor. Otherwise, a precondition evaluation error will be thrown.
runconfig:
  InputFilePath:
    L0_NEN_RST: null
  SPSName: "L0A_PGE"
  PGEName: "L0A_PGE"
  ProductPath: "/data/work/output"
  ShortName: "L0A"
  ProductCounter: null
  DebugSwitch: "Off"
  EchoToScreenSwitch: "Off"
  FullPathname: "/home/pgeuser/nisar_pge/l0a/nisar_l0a_processor.py"
  CompositeReleaseID: null
  ProcessingType: "PR"

  pge_job_name: null
  mock_metadata: null
  base_name: null

# This lists all the precondition evaluation steps that this PGE needs to run prior to running the PGE.
# These functions should be defined in the precondition_definition.yaml
preconditions:
  - get_crid
  - get_products
  - get_metadata
  - get_product_counter
  - set_product_time
  - set_pge_job_name
  - set_mock_metadata
  - set_base_name

# For any of the precondition evaluation steps listed in the preconditions area,
# specify function arguments here

# Looks at the context to get metadata
get_metadata:
  keys:
    - state_config_id

# Determines the product counter value to set in the Run Config
# When specifying a "value" field, it will just use the hardcoded value.
#TODO: See if we can determine this dynamically
get_product_counter:
  value: "001"

# This value will get used as part of the job id during the job submission step.
set_pge_job_name:
  template: ${state_config_id}

# When doing simulated PGE runs, this will mock together a set of metadata for the met.json output
set_mock_metadata:
  # Hard-coded values
  static:
    PGE_Name: "L0A"
    PGE_Version: "R0.0"
    Min_VCFC: 1
    Max_VCFC: 28
    Missing_VCFC_List:
      - 0
      - 3
      - 6
      - 9
      - 12
      - 15
      - 18
      - 21
      - 24
      - 27
    Missing_VCFC_ERT_List: "2019-10-29T17:13:05.813023Z, 2019-10-29T17:13:05.813123Z, 2019-10-29T17:13:05.813199Z,
    2019-10-29T17:13:05.813266Z, 2019-10-29T17:13:05.813342Z, 2019-10-29T17:13:05.813411Z"
    DAPHNE_Flag_Count: 0

  dynamic:
    # Keys represent the metadata keys that will go into the mock metadata
    # Values represent metadata that is assumed to exist in the job_params
    Input_Files: L0_NEN_RST
    ProductionDateTime: ProductionDateTime
    CompositeReleaseID: CompositeReleaseID
    ProductCounter: ProductCounter

# When performing simulated PGE runs, this sets the base name to be used when generating the output
# products.
set_base_name:
  template: NISAR_L0_PR_RRST_VC00_${DAPHNE_Min_Time_Tag}_${DAPHNE_Max_Time_Tag}_${CompositeReleaseID}_${ProductCounter}
  # This tells the function how to format the date time string in the file name
  date_time_formats:
    DAPHNE_Min_Time_Tag: "%Y%m%dT%H%M%S"
    DAPHNE_Max_Time_Tag: "%Y%m%dT%H%M%S"


# The PGE name
pge_name: "L0A"

# Set the primary input type here
primary_input: "L0_NEN_RST"

# List the extensions that the PGE generates
output_types:
  - bin
  - bin.log
  - bin.met

# List the groups that Chimera will need to localize
localize_groups:
  - InputFilePath


(lightbulb) Have Questions? Ask a HySDS Developer:

Anyone can join our public Slack channelto learn more about HySDS. JPL employees can join #HySDS-Community

(blue star)

JPLers can also ask HySDS questions atStack Overflow Enterprise

(blue star)

Live Search
placeholderSearch HySDS Wiki