This site is a collection of definitions and descriptions of HySDS concepts, tools and procedures.
see also: Beginner's Guide to HySDS
Jobs-
Jobs are subclasses of Tasks that add notions of running a sub-process executable in a working directory of a worker.
Tasks
Tasks are Celery-level python functions that are dispatched to run in the distributed workers.
Tasks are executed on workers.
Jobs are subclasses of Tasks that add notions of running a subprocess executable in a working directory of a worker.
Tasks have event states as they transition from queued to running and exit conditions.
Workers
- Puppet Modules
- Cluster Provisioning
- Cloud auto scaling
- Job Resiliency
- JPL tools
Verdi
Figaro
- 3rd party tools
Jenkins
Elastisearch
RabbitMQ
Celery
Python
Docker
Terraform
Supervisord
AWS
EC2
S3
- Faceted Searches
Datasets
Resources
Metrics
- SDS Watch
- Multi-tiered dataset caching