Versions Compared

Old Version 68

changes.mady.by.user vitrandao

Saved on

compared with

New Version 69

changes.mady.by.user Sehoon Park

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

SDSWatch

SDSWatch is designed to gain insight into what compute nodes are doing in real-time, and Kibana dashboard / visualization with the power of aggregating / faceting is used to achieve that purpose. Please click on this link if you’re interested in seeing what Kibana dashboard for SDSWatch looks like.

...

  1. PGE SDSWatch log

    a. These logs typically will be on verdi job worker: /data/work/job/<year>/<month>/<hour>/<minute>/<source_id>/<source_type>.pge.sdswatch.log

    b. Naming your sdswatch log as <source_type>.pge.sdswatch.log

    c. Format: <timestamp ISO 8601>, <key>, <value>

...

Client side (on compute node): Logtash with appropriate configuration will listen to sdswatch log files at two locations

  • /data/work/jobs/<year>/<month>/<day>/<hour>/<minute>/<job_id>/<job_type>.pge.sdswatch.log for pge logs.

  • /home/ops/verdi/log/<name>.sdswatch.log for any other sdswatch logs (currently only job worker) on the compute node

...

I first developed SDSWatch on hysdsops@100.67.35.12 hysdsops@<your-server-ip-address> for the server side and hysdsops@100.67.32.182 hysdsops@<your-client-ip-address> for the client side, so they may be still there.

On hysdsops@100.67.32.182 for the client sidehysdsops@<your-client-ip-address> for the client side, here is my work:

  • /export/home/hysdsops/verdi/share/sdswatch-client

  • /export/home/hysdsops/verdi/etc/logstash.conf

  • /export/home/hysdsops/verdi/etc/filebeat.yml (not in the current SDSWatch-client implementation, but you may find it helpful when migrating Logstash to Filebeat on the client side)

  • /export/home/hysdsops/verdi/etc/filebeat-configs (not in the current SDSWatch-client implementation, but you may find it helpful when migrating Logstash to Filebeat on the client side)

On hysdsops@100.67.35.12 a for the server side, here is my work:

...

To run SDSWatch client on hysdsops@100.67.32.182hysdsops@<your-client-ip-address>,

  • supervisor.d is already set up to run it automatically

  • check out section To check if the sdswatch-client is running correctly below.

To run SDSWatch server on hysdsops@100.67.35.12hysdsops@<your-server-ip-address>,

  • run “cd /export/home/hysdsops/tmp/sdswatch-server”, then run “bash sdswatch-server.sh”

  • Note: I used “chmod 777 -R data” for the data directory inside “sdswatch-server' since couldn’t find a way to make it work without using 777 mode 😞

  • After running 2 command lines above, here is the link to open Kibana dashboard: https://100.67.35.12<kibana-ip-address>:1502/app/kibana#/home?_g=()

Highly recommended: log in to both servers, play around with the current SDSWatch, and read all the code since there are not many. I recommend finding the corresponding files here https://github.com/hysds/hysds-sdswatch because there are more documentation. (tips: if you are reading the client side, find sdswatch-client.sh and read it first. If you are reading the server side, find sdswatch-server.sh and read it first. Ignore all relevant files to filebeat for now)

...

  • Go to https://github.com/hysds/hysds-sdswatch, open onserver directory and download all the files in it.

  • Log in to server hysdsops@100.67.35.12server hysdsops@<your-server-ip-address>

  • Copy the sdswatch-server directory from your machine and put it in /export/home/hysdsops/tmp/ in the server. Also, in the /export/home/hysdsops/tmp/sdswatch-server/, create an empty data directory to save Elasticsearch data.

  • To start SDSWatch server, on the server’s terminal

    • cd /export/home/hysdsops/tmp/sdswatch-server

    • give read and write permission for container to to write into /export/home/hysdsops/tmp/sdswatch-server/data directory (“chmod 777 -R data” works, but you cannot do it because of security reason. I couldn’t think of ways to make it work without “chmod 777 -R data”)

    • bash sdswatch-server.sh

  • You can access Kibana through localhost:5601 of the server.

...

  • Go to https://github.com/hysds/hysds-sdswatch, open onclient directory and download all the files in it.

  • Log in to server hysdsops@100.67.32.182hysdsops@<your-client-ip-address>

  • Copy the sdswatch-client directory from your machine and put it in /export/home/hysdsops/verdi/share/ in the server. Also, in the /export/home/hysdsops/verdi/share/sdswatch-client/, create an empty data directory to save Logstash history (Don’t delete this directory when there are a lot of old sdswatch logs on your system. If you delete this directory, Logstash will resend old logs).

  • Copy the logstash.conf file and put it in /export/home/hysdsops/verdi/etc/

  • Notice that there are a few filebeat files and directories. Don’t touch it for now. I’ll mention them later.

How to add sdswatch-sclient to supervisor.d so supervisor.d will automatically start Logstash when the server boots up again.

Note for hysdsops@100.67.32.182hysdsops@<your-client-ip-address>: I already did it, but you may find it helpful to read it again.

...

  • Understand how the client and server currently work first by playing around with it.

  • I recommend reading all the relevant files since there are not many (you should ignore all the Filebeat files on the client side for now) (tip: starting with the sdswatch-server.sh and sdswatch-client.sh). Try to understand the Logstash configuration file on the client side.

  • When migrating Logstash to Filebeat on the client side, the only thing you need to modify on the server side is the Logstash configuration (just adding a filter block between input block and output block).

  • I recommend trying out Filebeat on your local machine with SDSWatch logs first. Try using Filebeat to scoop up sdswatch logs and send it to console. Then look at the output logs printed in console, and investigate the fields inside the output logs. Then compare it with the assumed input in the current Logstash configuration in SDSWatch-client. Try to play around with Filebeat “add_field” feature and see if you can make the output logs from Filebeat to have the required information.

  • When you figure out how to make the output logs from Filebeat look right, check out filebeat.yml and filebeat-configs that I wrote that are currently in the system or on github. Using these files as a starting point. (Remember to always allow “live reloading” feature so we can always update it during production)

  • I also already wrote a configuration file to create docker container with Filebeat, you may want to log into hysdsops@100.67.32.182 hysdsops@<your-client-ip-address> and find the directory /export/home/hysdsops/verdi/share/filebeat-sdswatch-client. filebeat-sdswatch-client is similar to sdswatch-client but it’s for filebeat. However, when I ran it, there was an error that I couldn’t figure out why. This problem prevented me from migrating Logstash to Filebeat during my internship. Error when running Filebeat in docker: /usr/local/bin/docker-entrypoint: line 8: exec: filebeat: not found (I asked one of the people in Elastic forum, you may find it helpful)

  • The filebeat relevant files weren’t tested yet so read it with a grain of salt.

...