SciFlo Job

As briefly mentioned before SciFlo is a workflow framework. It uses a XML workflow definition document to know the sequence of steps. Each step of a workflow is known as a process. The workflow definition documents defines the flow of the processes and how to run each process. More about the workflow definition documents can be found here: SciFlo Workflow Definition

SciFlo Bash Script

Workflow jobs are executed by the run_sciflo.sh script (https://github.com/hysds/chimera/blob/develop/chimera/run_sciflo.sh).

It sets up the BASE_PATH, PYTHONPATH and other environment requirements for the SciFlo job.

It takes 2 positional arguments:

1. MODULE_PATH – Location to the adaptation module

2. WF_DIR – Path to the directory containing the workflow definition document

3. WF_NAME – Name of the Algorithm/Job/PGE to run

MODULE_PATH="$1"
WF_DIR="$2"
WF_NAME="$3"

It exports the adaptation module’s path to the python path so the scripts can resolve dependency imports.

export PYTHONPATH=${MODULE_PATH}:$PYTHONPATH

It constructs the location of the workflow definition document with WF_DIR and WF_NAME i.e. $WF_DIR/$WF_NAME.sf.xml

It runs the run_sciflo.py script with the required inputs. See section “SciFlo Python Script”.

python $BASE_PATH/run_sciflo.py $WF_DIR/$WF_NAME.sf.xml _context.json output > run_sciflo_$WF_NAME.log 2>&1
STATUS=$?
echo -n "Finished running $PGE run_sciflo.py: " 1>&2
date 1>&2
if [ $STATUS -ne 0 ]; then
  echo "Failed to run $PGE run_sciflo.py" 1>&2
  cat run_sciflo_$WF_NAME.log 1>&2
  echo "{}"
  exit $STATUS
fi

SciFlo Python Script

The run_sciflo.py is the python script that executes the SciFlo job.

It takes 3 positional arguments:

1. sfl_file – Workflow definition document - has the file extension .wf.xml

2. context_file – Job’s context file aka _context.json

3. output_folder – Directory to write outputs of the job i.e. output

This script imports the sciflo_util and calls the run_sciflo function.

from chimera.commons.sciflo_util import run_sciflo

def main(sfl_file, context_file, output_folder):
    sfl_file = os.path.abspath(sfl_file)
    context_file = os.path.abspath(context_file)
    output_file = os.path.abspath(output_folder)
    LOGGER.info("sfl_file: %s" % sfl_file)
    LOGGER.info("context_file: %s" % context_file)
    return run_sciflo(sfl_file, ["sf_context=%s" % context_file], output_folder)

The command to run SciFlo is $HOME/verdi/bin/sflExec.py -s -f -o output --args "sfl_context=_context.json", "pge.wf.xml"

In the sciflo_util.run_sciflo function:

This command is constructed and run.

# build paths to executables
    sflexec_path = os.path.join(
        os.environ['HOME'], 'verdi', 'bin', 'sflExec.py')

    # execute sciflo
    cmd = [sflexec_path, "-s", "-f", "-o", output_dir,
           "--args", '"%s"' % ','.join(sfl_args), sfl_file]
    print("Running sflExec.py command:\n%s" % ' '.join(cmd))
    status = os.system(' '.join(cmd))

Based on status of the SciFlo run it logs out the error and copies the files generated by the SciFlo run into the specified output directory.

    if status != 0:
        extract_error('%s/sciflo.json' % output_dir)
        status = 1

    # copy smap_sciflo work and exec dir
    try:
        copy_sciflo_work(output_dir)
    except Exception:
        pass