...

FROM centos:7

ARG HOME =/root
ARG VERSION="3.7.9"

WORKDIR $HOME

# RUN yum update -y && \
RUN yum install gcc openssl-devel bzip2-devel libffi-devel openldap-devel readline-devel make wget git -y && \
    cd /tmp && \
    # installing python 3
    wget https://www.python.org/ftp/python/3.7.9${VERSION}/Python-3.7.9${VERSION}.tgz && \
    tar xzf Python-3.7.9${VERSION}.tgz && \
    cd Python-3.7.9${VERSION} && \
    ./configure --enable-optimizations && \
    make altinstall && \
    ln -s /usr/local/bin/python3.7python${VERSION:0:3} /usr/local/bin/python3 && \
    ln -s /usr/local/bin/pip3.7pip${VERSION:0:3} /usr/local/bin/pip3 && \
    pip3 install --no-cache-dir --upgrade pip && \
    pip3 install --no-cache-dir gnureadline && \
    rm -f /tmp/Python-3.7.9${VERSION}.tgz && \
    rm -rf /tmp/Python-3.7.9${VERSION} && \
    # installing HySDS libraries
    cd $HOME && \
    git clone https://github.com/hysds/prov_es.git && \
    git clone https://github.com/hysds/osaka.git && \
    git clone https://github.com/hysds/hysds_commons.git && \
    git clone https://github.com/hysds/hysds.git && \
    pip3 install --no-cache-dir -e prov_es/ && \
    pip3 install --no-cache-dir -e osaka/ && \
    pip3 install --no-cache-dir -e hysds_commons/ && \
    pip3 install --no-cache-dir -e hysds/ && \
    yum clean all && \
    rm -rf /var/cache/yum && \
    rm -r /tmp/*

WORKDIR $HOME
CMD ["/bin/bash"]

...

• Mozart

  • Mozart rest API

  • Elasticsearch

  • Logstash

  • Redis

  • RabbitMQ

  • Celery workers (wrapped with supervisord)

    • orchestrator (job validation)

    • etc.

• GRQ

  • GRQ2 rest API

  • Pele rest API

  • Elasticsearch

  • aws-es-proxy (if we’re using AWS ES for GRQ)

• Factotum

  • Celery workers

    • user_rules (jobs & datasets) processor

    • factotum job worker (small & large)

    • etc.

• Metrics

  • Redis

  • Logstash

  • Elasticsearch

  • Kibana

  • sdswatch

• CI

  • Jenkins

Before HySDS would run its services in their respective machine/instance (Mozart, GRQ, Metrics & Factotum) but moving to a k8 deployment will get rid off that as the engine will determine which k8 node will run which service

...

Stateful Application(s)

• Elasticsearch: https://sematext.com/blog/kubernetes-elasticsearch/

• Redis

• RabbitMQ

stateful applications save client data and deployments are more complicated

...

• it keeps the ordering of pods in your deployment so that it can accommodate a leader/worker architecture

• example provided by RabbitMQ’s documentation: https://github.com/rabbitmq/diy-kubernetes-examples/blob/master/gke/statefulset.yaml

Another option they recommend is to move your stateful applications into cloud managed services

• Redis is supported by AWS Elasticache, Azure Cache & GCP MemoryStore

• RabbitMQ supported by Amazon MQ, not sure if Azure or GCP support this

• AWS Elasticsearch service (but adds complexity as every request will have to be signed)

  • Elastic on Azure, but not sure how it works

Helm

Helm is a plugin (similar to homebrew, yum) and repository for kubernetes which hosts k8 “packages” and …

...

• it was written under the design of SSHing into other HySDS components (grq, factotum & metrics) and running commands such as pip install, etc.

• it relies on fabdric to copy files from mozart to other HySDS components

  • for example, sds update grq will clear out ~/sciflo/ops/ and copy over all the necessary files/repos from ~/mozart/ops/ to grq

  • can copy files from pod -> pod (kubectl cp my-pod:my-file my-file) but it can potentially mess things up

• this will not work with k8 b/c every service is completely de-coupled and in their own environment

• sds [start|stop|reset] [mozart|grq|metrics|factotum] will become somewhat obsolete (in its current state) because there there’s no need for supervisord to run its services

  • services will be running in their own standalone pod(s)

  • instead will use kubectl to manage the k8 services

  • sueprvisord may be used in the k8 pod for celery workers

    • b/c we have many celery workers (user_rules processing, orchestrator, etc), wrapping it in supervisord in a pod may clean things up

• will need to see how sds ship will be affected by kubernetes