• Created by Hook Hua, last modified by Lan Dang on May 05, 2017

Background

Mozart is a resource management service that enables managing jobs across a hybrid cloud computing resources.

Compute Nodes

Each compute node runs a worker that communicates back with Mozart.

Jobs, Tasks, and Workers

Tasks are Celery-level python functions that are dispatched to run in the distributed workers.

Workers are Celery-level workers that run tasks. Since jobs are tasks, they also run jobs within the context of a unique working directory.

Jobs are subclasses of Tasks that add notions of running a sub-process executable in a working directory of a worker.

Event Handling

On the compute nodes, supervisord ensures that Celery workers are running. Each worker celery uses RabbitMQ to community back to mozart the job queue and comm, as well as celery-level events (worker events and task events). Inside worker.py, it uses redis to send back job-level events.

Components

Mozart integrates together the following components:

Celery

Distributed Task Queue

Celery is a simple, flexible and reliable distributed system to process vast amounts of messages, while providing operations with the tools required to maintain such a system.

http://www.celeryproject.org/

http://docs.celeryproject.org/en/latest/

Flower

Real-time monitor and web admin for Celery distributed task queue.

https://github.com/mher/flower

http://flower.readthedocs.org/

Jobtastic

http://policystat.github.io/jobtastic/

RabbitMQ

RabbitMQ is the most widely deployed open source message broker

http://www.rabbitmq.com/

Redis

Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker. It supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs and geospatial indexes with radius queries. Redis has built-in replication, Lua scripting, LRU eviction, transactions and different levels of on-disk persistence, and provides high availability via Redis Sentinel and automatic partitioning with Redis Cluster.

https://redis.io/

ElasticSearch

Elasticsearch is a distributed, RESTful search and analytics engine capable of solving a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data so you can discover the expected and uncover the unexpected.

https://www.elastic.co/products/elasticsearch

Kibana

Kibana lets you visualize your Elasticsearch data and navigate the Elastic Stack, so you can do anything from learning why you're getting paged at 2:00 a.m. to understanding the impact rain might have on your quarterly numbers.

https://www.elastic.co/products/kibana

Logstash

Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite “stash.” (Ours is Elasticsearch, naturally.)

https://www.elastic.co/products/logstash

FacetView

FacetView is a pure javascript frontend for ElasticSearch search indices.

https://github.com/okfn/facetview