To leverage the computing power of pleiades, we port verdi to run bare-metal on pleiades and have it communicate with the HySDS cluster (a.k.a. PCM) on the Amazon Cloud to obtain jobs and submit results via the queues between the AWS Cloud and pleiades. The Docker virtualization system is considered unsafe for the shared supercomputer environment and therefore is not supported on pleiades. So verdi in its Docker form cannot be run directly on pleiades. Furthermore, all the PGEs in Docker containers have to be converted to singularity, another virtualization system that is allowed on pleiades. The porting of ARIA HySDS to pleiades consists of two major parts: (1) The container-builder. The Jenkins pipeline running on the ci (continuous integration) server triggers the new container-builder to, in addition to building the PGEs in the form of Docker containers, convert the Docker container into a singularity container, build a sandbox from the singularity container, and upload the singularity sandbox to the S3 storage; and (2) job_worker. The singularity extension dequeues jobs from Mozart on AWS, downloads the PGE in the form of a singularity sandbox from S3, forms the singularity exec command line, runs the PGE, and submit the results to GRQ on AWS.

Running HySDS Verdi Job Worker on Pleiades

...

Task handler raised error: WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM).')

Adaptations of HySDS/Pleiades

ARIA HySDS running Sentinel-1A/B S1-GUNW processing on Pleiades