Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


Page Navigation:

Table of Contents


(blue star) Confidence Level TBD  This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it.

Introduction

When we use Amazon Web Services autoscaling, we need to configure an alarm that is triggered when a certain condition is met.  Usually, this involves checking whether a CloudWatch metric falls within or outside a certain range. 

Since we want to trigger autoscaling when we have a large number of jobs in the system, Mozart needs to communicate this metric to Amazon, so it can be used in CloudWatch.   We can define custom metrics.   This is done via https://github.jpl.nasa.gov/hysds-org/hysds/blob/master/scripts/sync_ec2_job_metric.py

The program runs in  supervisord on Mozart.  This is not enabled by default, because it is customized based on which queues are being monitored.   You need to add a new program block for each queue you are monitoring.


This is the templated version kept in GitHub for ARIA Ops:

hysds_cluster_fab/files/supervisord.conf.mozart


Code

[program:sync_ec2_job_metric-{{ MONITORED_QUEUE_5 }}]

directory=/home/ops/mozart/ops/hysds/scripts

command=/home/ops/mozart/ops/hysds/scripts/sync_ec2_job_metric.py --interval 60 {{ MONITORED_QUEUE_5 }}

process_name=%(program_name)s

priority=1

numprocs=1

numprocs_start=0

redirect_stderr=true

stdout_logfile=%(here)s/../log/%(program_name)s.log

startsecs=10



MONITORED_QUEUE_5 is a template variable defined in hysds_cluster_fab/context.sh


export MONITORED_QUEUE=spyddder-extract



Once Fabric pushes the configuration to Mozart, it can be found on Mozart at:  /home/ops/mozart/etc/supervisord.conf


[program:sync_ec2_job_metric-spyddder-extract]

directory=/home/ops/mozart/ops/hysds/scripts

command=/home/ops/mozart/ops/hysds/scripts/sync_ec2_job_metric.py --interval 60 spyddder-extract

process_name=%(program_name)s

priority=1

numprocs=1

numprocs_start=0

redirect_stderr=true

stdout_logfile=%(here)s/../log/%(program_name)s.log

startsecs=10



(lightbulb) Have Questions? Ask a HySDS Developer:

Anyone can join our public Slack channelto learn more about HySDS. JPL employees can join #HySDS-Community

(blue star)

JPLers can also ask HySDS questions atStack Overflow Enterprise

(blue star)