SciFlo Job

As briefly mentioned before SciFlo is a workflow framework. It uses a XML workflow definition document to know the sequence of steps. Each step of a workflow is known as a process. The workflow definition documents defines the flow of the processes and how to run each process. More about the workflow definition documents can be found here: SciFlo Workflow Definition

SciFlo Bash Script

Any workflow job is executed by the run_sciflo.sh script.

It first sets up the BASE_PATH, PYTHONPATH and other environment requirements for the sciflo job.

It then reads in the inputs mentioned below:

1. Location to the adaptation module

2. Path to the directory containing the workflow definition document

3. Name of the Algorithm/Job/PGE to run

MODULE_PATH="$1"
WF_DIR="$2"
WF_NAME="$3"

It exports the adaptation module’s path to the python path so the scripts can resolve dependency imports.

export PYTHONPATH=${MODULE_PATH}:$PYTHONPATH

It constructs the location of the workflow definition document with WF_DIR and WF_NAME i.e. $WF_DIR/$WF_NAME.sf.xml

It runs the run_sciflo.py script with the required inputs.

python $BASE_PATH/run_sciflo.py $WF_DIR/$WF_NAME.sf.xml _context.json output > run_sciflo_$WF_NAME.log 2>&1
STATUS=$?
echo -n "Finished running $PGE run_sciflo.py: " 1>&2
date 1>&2
if [ $STATUS -ne 0 ]; then
  echo "Failed to run $PGE run_sciflo.py" 1>&2
  cat run_sciflo_$WF_NAME.log 1>&2
  echo "{}"
  exit $STATUS
fi

SciFlo Python Script

The run_sciflo.py is the python script that executes the sciflo job. It takes 2 positional arguments:

1. Workflow definition document - has the file extension .wf.xml

2. Job’s context file aka _context.json

3. Directory to write outputs of the job i.e. output

This script imports the sciflo_util and calls the run_sciflo function.

from chimera.commons.sciflo_util import run_sciflo

def main(sfl_file, context_file, output_folder):
    sfl_file = os.path.abspath(sfl_file)
    context_file = os.path.abspath(context_file)
    output_file = os.path.abspath(output_folder)
    LOGGER.info("sfl_file: %s" % sfl_file)
    LOGGER.info("context_file: %s" % context_file)
    return run_sciflo(sfl_file, ["sf_context=%s" % context_file], output_folder)

The command to run sciflo is $HOME/verdi/bin/sflExec.py -s -f -o output --args "sfl_context=_context.json", "pge.wf.xml"

In the sciflo_util.run_sciflo function:

This command is constructed and run.

# build paths to executables
    sflexec_path = os.path.join(
        os.environ['HOME'], 'verdi', 'bin', 'sflExec.py')

    # execute sciflo
    cmd = [sflexec_path, "-s", "-f", "-o", output_dir,
           "--args", '"%s"' % ','.join(sfl_args), sfl_file]
    print("Running sflExec.py command:\n%s" % ' '.join(cmd))
    status = os.system(' '.join(cmd))

Based on status of the Sciflo run it logs out the error and copies the files generated by the sciflo run into the specified output directory.

    if status != 0:
        extract_error('%s/sciflo.json' % output_dir)
        status = 1

    # copy smap_sciflo work and exec dir
    try:
        copy_sciflo_work(output_dir)
    except Exception:
        pass