PGE Config File
Confidence Level Moderate This article includes input from several JPLers. Multiple subject matter experts can indicate that a page is more frequently reviewed and updated. |
|---|
This YAML file is the key that drives what preconditions are evaluated, what kind of job is submitted and what post processing steps need to be run.
A PGE config file consists of the following elements:
Runconfig template:
The runconfig section is intended to be used for dynamically generating the runconfig file used when running the NISAR PGEs. Any values set with a "null" should be filled in by the input preprocessor. Otherwise, a precondition evaluation error will be thrown.Precondition functions to run:
This lists all the precondition evaluation steps that this PGE needs to run prior to running the PGE.additional information for precondition functions (if required)
For any of the precondition evaluation steps listed in the preconditions area, we specify function arguments.PGE name:
name of the PGEPrimary input to PGE:
specify primary dataset type that’s input to the PGEPGE Output file types:
List the extensions that the PGE generatesFiles to be downloaded by PGE execution job:
List the groups within the run config that the PGE job will need to localizeMock metadata (if PGE run is being mocked):
When doing simulated PGE runs, this will mock together a set of metadata for the met.json output.PGE Output file’s base name (if PGE run is being mocked):
When performing simulated PGE runs, this sets the base name to be used when generating the output products.
Here is an example PGE config:
# The runconfig section is intended to be used for dynamically generating the runconfig file used
# when running the NISAR PGEs. Any values set with a "null" should be filled in by the input
# preprocessor. Otherwise, a precondition evaluation error will be thrown.
runconfig:
InputFilePath:
L0_NEN_RST: null
SPSName: "L0A_PGE"
PGEName: "L0A_PGE"
ProductPath: "/data/work/output"
ShortName: "L0A"
ProductCounter: null
DebugSwitch: "Off"
EchoToScreenSwitch: "Off"
FullPathname: "/home/pgeuser/nisar_pge/l0a/nisar_l0a_processor.py"
CompositeReleaseID: null
ProcessingType: "PR"
pge_job_name: null
mock_metadata: null
base_name: null
# This lists all the precondition evaluation steps that this PGE needs to run prior to running the PGE.
# These functions should be defined in the precondition_definition.yaml
preconditions:
- get_crid
- get_products
- get_metadata
- get_product_counter
- set_product_time
- set_pge_job_name
- set_mock_metadata
- set_base_name
# For any of the precondition evaluation steps listed in the preconditions area,
# specify function arguments here
# Looks at the context to get metadata
get_metadata:
keys:
- state_config_id
# Determines the product counter value to set in the Run Config
# When specifying a "value" field, it will just use the hardcoded value.
#TODO: See if we can determine this dynamically
get_product_counter:
value: "001"
# This value will get used as part of the job id during the job submission step.
set_pge_job_name:
template: ${state_config_id}
# When doing simulated PGE runs, this will mock together a set of metadata for the met.json output
set_mock_metadata:
# Hard-coded values
static:
PGE_Name: "L0A"
PGE_Version: "R0.0"
Min_VCFC: 1
Max_VCFC: 28
Missing_VCFC_List:
- 0
- 3
- 6
- 9
- 12
- 15
- 18
- 21
- 24
- 27
Missing_VCFC_ERT_List: "2019-10-29T17:13:05.813023Z, 2019-10-29T17:13:05.813123Z, 2019-10-29T17:13:05.813199Z,
2019-10-29T17:13:05.813266Z, 2019-10-29T17:13:05.813342Z, 2019-10-29T17:13:05.813411Z"
DAPHNE_Flag_Count: 0
dynamic:
# Keys represent the metadata keys that will go into the mock metadata
# Values represent metadata that is assumed to exist in the job_params
Input_Files: L0_NEN_RST
ProductionDateTime: ProductionDateTime
CompositeReleaseID: CompositeReleaseID
ProductCounter: ProductCounter
# When performing simulated PGE runs, this sets the base name to be used when generating the output
# products.
set_base_name:
template: NISAR_L0_PR_RRST_VC00_${DAPHNE_Min_Time_Tag}_${DAPHNE_Max_Time_Tag}_${CompositeReleaseID}_${ProductCounter}
# This tells the function how to format the date time string in the file name
date_time_formats:
DAPHNE_Min_Time_Tag: "%Y%m%dT%H%M%S"
DAPHNE_Max_Time_Tag: "%Y%m%dT%H%M%S"
# The PGE name
pge_name: "L0A"
# Set the primary input type here
primary_input: "L0_NEN_RST"
# List the extensions that the PGE generates
output_types:
- bin
- bin.log
- bin.met
# List the groups that Chimera will need to localize
localize_groups:
- InputFilePath
Have Questions? Ask a HySDS Developer: |
Anyone can join our public Slack channel to learn more about HySDS. JPL employees can join #HySDS-Community
|
JPLers can also ask HySDS questions at Stack Overflow Enterprise
|