CloudWatch custom metrics

Created by Lan Dang on Aug 02, 2016

Page Navigation:

Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it.

When we use Amazon Web Services autoscaling, we need to configure an alarm that is triggered when a certain condition is met. Usually, this involves checking whether a CloudWatch metric falls within or outside a certain range.

Since we want to trigger autoscaling when we have a large number of jobs in the system, Mozart needs to communicate this metric to Amazon, so it can be used in CloudWatch. We can define custom metrics. This is done via https://github.jpl.nasa.gov/hysds-org/hysds/blob/master/scripts/sync_ec2_job_metric.py

The program runs in supervisord on Mozart. This is not enabled by default, because it is customized based on which queues are being monitored. You need to add a new program block for each queue you are monitoring.

This is the templated version kept in GitHub for ARIA Ops:

hysds_cluster_fab/files/supervisord.conf.mozart

[program:sync_ec2_job_metric-{{ MONITORED_QUEUE_5 }}]

directory=/home/ops/mozart/ops/hysds/scripts

command=/home/ops/mozart/ops/hysds/scripts/sync_ec2_job_metric.py --interval 60 {{ MONITORED_QUEUE_5 }}

process_name=%(program_name)s

priority=1

numprocs=1

numprocs_start=0

redirect_stderr=true

stdout_logfile=%(here)s/../log/%(program_name)s.log

startsecs=10