...
Can run qstat -f $PBS_JOBID and compare the output for resources_used.walltime and Resources_List.walltime to see how much time remains. If insufficient time, then the pbs wrapper script can sigterm gracefully the PID of verdi job worker to trigger verdi job worker to exit the process and therefore end the PBS node.
Based on current metrics, topsapp-singularity generating one S1-GUNW takes a mean of 1.5 hours on Broadwell. So add some margin and use that as max. e.g. requiring at least 2-hours remaining for current PBS job. if not, then exit the PBS job.
Lei.Pan@jpl.nasa.gov (Unlicensed) TODO
...