SciFlo Job
- Namrata Malarout
- Christopher Rivas
- Topher Allen
Page Navigation: |
|---|
ย
Confidence Level Highย This articleย been formally reviewed and is signed off on by a relevant subject matter expert.ย |
|---|
As briefly mentioned before SciFlo is a workflow framework. It uses a XML workflow definition document to know the sequence of steps. Each step of a workflow is known as a process. The workflow definition documents defines the flow of the processes and how to run each process. More about the workflow definition documents can be found here: SciFlo Workflow Definition
ย
SciFlo Bash Script
Workflow jobs are executed by the run_sciflo.sh script (chimera/chimera/run_sciflo.sh at develop ยท hysds/chimera).
It sets up the BASE_PATH, PYTHONPATH and other environment requirements for the SciFlo job.
It takes 3 positional arguments:
MODULE_PATHโ Location to the adaptation moduleWF_DIRโ Path to the directory containing the workflow definition documentWF_NAMEโ Name of the Algorithm/Job/PGE to run
MODULE_PATH="$1"
WF_DIR="$2"
WF_NAME="$3"It exports the adaptation moduleโs path to the python path so the scripts can resolve dependency imports.
export PYTHONPATH=${MODULE_PATH}:$PYTHONPATHIt constructs the location of the workflow definition document with WF_DIR and WF_NAME i.e. $WF_DIR/$WF_NAME.sf.xml
It runs the run_sciflo.py script with the required inputs. See section โSciFlo Python Scriptโ.
python $BASE_PATH/run_sciflo.py $WF_DIR/$WF_NAME.sf.xml _context.json output > run_sciflo_$WF_NAME.log 2>&1
STATUS=$?
echo -n "Finished running $PGE run_sciflo.py: " 1>&2
date 1>&2
if [ $STATUS -ne 0 ]; then
echo "Failed to run $PGE run_sciflo.py" 1>&2
cat run_sciflo_$WF_NAME.log 1>&2
echo "{}"
exit $STATUS
fiSciFlo Python Script
The run_sciflo.py is the python script that executes the SciFlo job.
It takes 3 positional arguments:
sfl_fileโ Workflow definition document - has the file extension.wf.xmlcontext_fileโ Jobโs context file aka_context.jsonoutput_folderโ Directory to write outputs of the job i.e.output
This script imports the sciflo_util and calls the run_sciflo function.
from chimera.commons.sciflo_util import run_sciflo
def main(sfl_file, context_file, output_folder):
sfl_file = os.path.abspath(sfl_file)
context_file = os.path.abspath(context_file)
output_file = os.path.abspath(output_folder)
LOGGER.info("sfl_file: %s" % sfl_file)
LOGGER.info("context_file: %s" % context_file)
return run_sciflo(sfl_file, ["sf_context=%s" % context_file], output_folder)The command to run SciFlo is $HOME/verdi/bin/sflExec.py -s -f -o output --args "sfl_context=_context.json", "pge.wf.xml"
In the sciflo_util.run_sciflo function:
This command is constructed and run.
# build paths to executables
sflexec_path = os.path.join(
os.environ['HOME'], 'verdi', 'bin', 'sflExec.py')
# execute sciflo
cmd = [sflexec_path, "-s", "-f", "-o", output_dir,
"--args", '"%s"' % ','.join(sfl_args), sfl_file]
print("Running sflExec.py command:\n%s" % ' '.join(cmd))
status = os.system(' '.join(cmd))Based on status of the SciFlo run it logs out the error and copies the files generated by the SciFlo run into the specified output directory.
if status != 0:
extract_error('%s/sciflo.json' % output_dir)
status = 1
# copy smap_sciflo work and exec dir
try:
copy_sciflo_work(output_dir)
except Exception:
passย
Have Questions? Ask a HySDS Developer: |
Anyone can join our public Slack channel to learn more about HySDS. JPL employees can join #HySDS-Community
|
JPLers can also ask HySDS questions at Stack Overflow Enterprise
|
Page Information: |
|---|
Was this page useful? |
Contribution History:
|
Subject Matter Expert: @Namrata Malarout |
Find an Error? Is this document outdated or inaccurate? Please contact the assigned Page Maintainer: @Namrata Malarout |