HySDS Intro: Reference Materials

This site is a collection of definitions and descriptions of HySDS concepts, tools and procedures.

see also: Beginner's Guide to HySDS

• FAQs
• Glossary
• Container Technology

Jobs-

Jobs are subclasses of Tasks that add notions of running a sub-process executable in a working directory of a worker.

Tasks

Tasks are Celery-level python functions that are dispatched to run in the distributed workers.

Tasks are executed on workers.

Jobs are subclasses of Tasks that add notions of running a subprocess executable in a working directory of a worker.

Tasks have event states as they transition from queued to running and exit conditions.

Workers

• Puppet Modules
• Cluster Provisioning
• Cloud auto scaling
• Job Resiliency
• JPL tools

Mozart

Verdi

Figaro

• 3rd party tools

Jenkins

Elastisearch

RabbitMQ

Celery

Python

Docker

Terraform

Supervisord

AWS

EC2

S3

• Faceted Searches

Datasets

Resources

Metrics

• SDS Watch
• Multi-tiered dataset caching