RabbitMQ Overview
RabbitMQ is used in combination with Celery to queue jobs. Logging into RabbitMQ can give you a broader view on the state of the system. Some examples where RabbitMQ can be useful is in determining why a job has been queued for a long time, determining if a job is in the wrong queue, or receiving real-time status of jobs. While Mozart can show the status of a job, it determines this through the data in elasticsearch. If for any reason elasticsearch wasn’t updated properly, Mozart may give you an incorrect status.
Logging in to RabbitMQ
The credentials for RabbitMQ will likely be located in a .netrc file in the Mozart Machine
Using RabbitMQ
Go to the Queues Tab
Type ^(?!celery) under Filter and check Regex to rid of celery related queues.
Messages
Ready: Jobs are in the queue ready to be run by a worker
Unacked: A worker started a job but hasn’t necessarily finished. There is often a timeout in the case that a job remains in an unacked state for a set time period.
Total:
State
Running: A worker is listening to the queue
Idle: A worker is not listening to the queue. May be due to network and connectivity issues or because the queue doesn’t actually exist.
Using RabbitMQ for Debugging
When jobs are in a Ready state but not an Unacked state, it can mean that the Autoscaling Group isn’t working correctly, or that a worker isn’t listening in on the queue.