Components

HySDS integrates together the following 3rd Party services and software:

Celery: Distributed Task Queue

Celery is a simple, flexible and reliable distributed system to process vast amounts of messages, while providing operations with the tools required to maintain such a system.

http://www.celeryproject.org/

http://docs.celeryproject.org/en/latest/

Flower

Real-time monitor and web admin for Celery distributed task queue.

https://github.com/mher/flower

http://flower.readthedocs.org/

Jobtastic

http://policystat.github.io/jobtastic/

RabbitMQ

RabbitMQ is the most widely deployed open source message broker

http://www.rabbitmq.com/

Redis

Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker. It supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs and geospatial indexes with radius queries. Redis has built-in replication, Lua scripting, LRU eviction, transactions and different levels of on-disk persistence, and provides high availability via Redis Sentinel and automatic partitioning with Redis Cluster.

https://redis.io/

ElasticSearch

Elasticsearch is a distributed, RESTful search and analytics engine capable of solving a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data so you can discover the expected and uncover the unexpected.

https://www.elastic.co/products/elasticsearch

Kibana

Kibana lets you visualize your Elasticsearch data and navigate the Elastic Stack, so you can do anything from learning why you're getting paged at 2:00 a.m. to understanding the impact rain might have on your quarterly numbers.

https://www.elastic.co/products/kibana

Logstash

Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite “stash.” (Ours is Elasticsearch, naturally.)

https://www.elastic.co/products/logstash

FacetView

FacetView is a pure javascript frontend for ElasticSearch search indices.

https://github.com/okfn/facetview

Jenkins

Terraform

Flask

Anaconda (Python3)

Fabric