PGE Config File

Confidence Level Moderate This article includes input from several JPLers. Multiple subject matter experts can indicate that a page is more frequently reviewed and updated.

Confidence Level Moderate This article includes input from several JPLers. Multiple subject matter experts can indicate that a page is more frequently reviewed and updated.

This YAML file is the key that drives what preconditions are evaluated, what kind of job is submitted and what post processing steps need to be run.

A PGE config file consists of the following elements:

Runconfig template:
The runconfig section is intended to be used for dynamically generating the runconfig file used when running the NISAR PGEs. Any values set with a "null" should be filled in by the input preprocessor. Otherwise, a precondition evaluation error will be thrown.
Precondition functions to run:
This lists all the precondition evaluation steps that this PGE needs to run prior to running the PGE.
additional information for precondition functions (if required)
For any of the precondition evaluation steps listed in the preconditions area, we specify function arguments.
PGE name:
name of the PGE
Primary input to PGE:
specify primary dataset type that’s input to the PGE
PGE Output file types:
List the extensions that the PGE generates
Files to be downloaded by PGE execution job:
List the groups within the run config that the PGE job will need to localize
Mock metadata (if PGE run is being mocked):
When doing simulated PGE runs, this will mock together a set of metadata for the met.json output.
PGE Output file’s base name (if PGE run is being mocked):
When performing simulated PGE runs, this sets the base name to be used when generating the output products.

Here is an example PGE config:

# The runconfig section is intended to be used for dynamically generating the runconfig file used
# when running the NISAR PGEs. Any values set with a "null" should be filled in by the input
# preprocessor. Otherwise, a precondition evaluation error will be thrown.
runconfig:
  InputFilePath:
    L0_NEN_RST: null
  SPSName: "L0A_PGE"
  PGEName: "L0A_PGE"
  ProductPath: "/data/work/output"
  ShortName: "L0A"
  ProductCounter: null
  DebugSwitch: "Off"
  EchoToScreenSwitch: "Off"
  FullPathname: "/home/pgeuser/nisar_pge/l0a/nisar_l0a_processor.py"
  CompositeReleaseID: null
  ProcessingType: "PR"

  pge_job_name: null
  mock_metadata: null
  base_name: null

# This lists all the precondition evaluation steps that this PGE needs to run prior to running the PGE.
# These functions should be defined in the precondition_definition.yaml
preconditions:
  - get_crid
  - get_products
  - get_metadata
  - get_product_counter
  - set_product_time
  - set_pge_job_name
  - set_mock_metadata
  - set_base_name

# For any of the precondition evaluation steps listed in the preconditions area,
# specify function arguments here

# Looks at the context to get metadata
get_metadata:
  keys:
    - state_config_id

# Determines the product counter value to set in the Run Config
# When specifying a "value" field, it will just use the hardcoded value.
#TODO: See if we can determine this dynamically
get_product_counter:
  value: "001"

# This value will get used as part of the job id during the job submission step.
set_pge_job_name:
  template: ${state_config_id}

# When doing simulated PGE runs, this will mock together a set of metadata for the met.json output
set_mock_metadata:
  # Hard-coded values
  static:
    PGE_Name: "L0A"
    PGE_Version: "R0.0"
    Min_VCFC: 1
    Max_VCFC: 28
    Missing_VCFC_List:
      - 0
      - 3
      - 6
      - 9
      - 12
      - 15
      - 18
      - 21
      - 24
      - 27
    Missing_VCFC_ERT_List: "2019-10-29T17:13:05.813023Z, 2019-10-29T17:13:05.813123Z, 2019-10-29T17:13:05.813199Z,
    2019-10-29T17:13:05.813266Z, 2019-10-29T17:13:05.813342Z, 2019-10-29T17:13:05.813411Z"
    DAPHNE_Flag_Count: 0

  dynamic:
    # Keys represent the metadata keys that will go into the mock metadata
    # Values represent metadata that is assumed to exist in the job_params
    Input_Files: L0_NEN_RST
    ProductionDateTime: ProductionDateTime
    CompositeReleaseID: CompositeReleaseID
    ProductCounter: ProductCounter

# When performing simulated PGE runs, this sets the base name to be used when generating the output
# products.
set_base_name:
  template: NISAR_L0_PR_RRST_VC00_${DAPHNE_Min_Time_Tag}_${DAPHNE_Max_Time_Tag}_${CompositeReleaseID}_${ProductCounter}
  # This tells the function how to format the date time string in the file name
  date_time_formats:
    DAPHNE_Min_Time_Tag: "%Y%m%dT%H%M%S"
    DAPHNE_Max_Time_Tag: "%Y%m%dT%H%M%S"


# The PGE name
pge_name: "L0A"

# Set the primary input type here
primary_input: "L0_NEN_RST"

# List the extensions that the PGE generates
output_types:
  - bin
  - bin.log
  - bin.met

# List the groups that Chimera will need to localize
localize_groups:
  - InputFilePath

Related Articles:
Page: Deploy PGE's onto Cluster Page: How to run a Matlab-based PGE Job Page: Input Preprocessor Page: PGE Config File Page: PGE Execution Page: PGE Jobs Page: Post Processor Page: Python 2 to Python 3 Page: Settings File

Have Questions? Ask a HySDS Developer:

Anyone can join our public Slack channel to learn more about HySDS. JPL employees can join #HySDS-Community

JPLers can also ask HySDS questions at Stack Overflow Enterprise

Page Information:

Was this page useful?

Yes No

Contribution History:

Topher Allen (1539 days ago)
Namrata Malarout (1620 days ago)

Subject Matter Expert:

@Namrata Malarout

Find an Error?

Is this document outdated or inaccurate? Please contact the assigned Page Maintainer:

@Lan Dang