Page Comparison

RabbitMQ Overview

RabbitMQ is used in combination with Celery to queue jobs. Logging into RabbitMQ can give you a broader view on the state of the system. Some examples where RabbitMQ can be useful is in determining why a job has been queued for a long time, determining if a job is in the wrong queue, or receiving real-time status of jobs. While Mozart can show the status of a job, it determines this through the data in elasticsearch. If for any reason elasticsearch wasn’t updated properly, Mozart may give you an incorrect status.

Logging in to RabbitMQ

The credentials for RabbitMQ will likely be located in a .netrc file in the Mozart Machine

Using RabbitMQ

1. Go to the Queues Tab

2. Type ^(?!celery) under Filter and check Regex to rid of celery related queues.

Image Modified

Messages

• Ready: Jobs are in the queue ready to be run by a worker

• Unacked: A worker started a job but hasn’t necessarily finished. There is often a timeout in the case that a job remains in an unacked state for a set time period.

• Total:

State

• Running: A worker is listening to the queue

• Idle: A worker is not listening to the queue. May be due to network and connectivity issues or because the queue doesn’t actually exist.

Using RabbitMQ for Debugging

• When jobs are in a Ready state but not an Unacked state, it can mean that the Autoscaling Group isn’t working correctly, or that a worker isn’t listening in on the queue.