How to debug on an ASG worker node


Confidence Level Moderate  This article includes input from several JPLers. Multiple subject matter experts can indicate that a page is more frequently reviewed and updated.

Confidence Level Moderate  This article includes input from several JPLers. Multiple subject matter experts can indicate that a page is more frequently reviewed and updated.

 

TODO: @Xing, @Marjorie Lucas @alphan, @Alexander Torres

Developer Considerations

  1. The node is still subject to spot terminations, so don't depend on it staying up forever

  2. The developer is now responsible for the life cycle of this node, to turn it off when it's not use and to terminate it when it's no longer needed

  3. The developer should probably relabel the node, if possible, so other developers know who is responsible for that instance, if any issues crop up.

  4. What is the impact on the operator, who may see this node in metrics and figaro?

Key Steps

(1) detach from ASG

  1. Go to the AWS EC2 console

  2. Click on the Auto Scaling Groups link on the left hand side column (usually towards the bottom)

  3. Click on the autoscaling group that your Verdi worker instance is in.

    1. After you click on the autoscaling group, a menu should show up on the bottom of the screen.

  4. Click on the Instance Management tab and locate your Verdi worker under Instances.

  5. Right click the instance you want to detach and click detach.

(2) disable harikiri

  1. Create a file with touch /data/work/.harikiri. Harikiri will not terminate the instance as long as such a file exists.

(3) docker run bash

  1. ssh into worker node

  2. cd into the work directory

  3. Edit the _run.sh script

    1. Edit the block of code starting with docker run ...

    2. append -it just after “docker run”

    3. remove PGE command (/home/ops/...) at the end of the docker run ... block and add /bin/bash

  4. Save _run.sh changes

  5. Run _run.sh

 

 

 


Related Articles:

Have Questions? Ask a HySDS Developer:

Anyone can join our public Slack channel to learn more about HySDS. JPL employees can join #HySDS-Community

JPLers can also ask HySDS questions at Stack Overflow Enterprise

Search HySDS Wiki

Page Information:

Was this page useful?

Yes No

Contribution History:

Subject Matter Expert:

@Alexander Torres

@Marjorie Lucas

Find an Error?

Is this document outdated or inaccurate? Please contact the assigned Page Maintainer:

@Alexander Torres

Note: JPL employees can also get answers to HySDS questions at Stack Overflow Enterprise: