HySDS core hackathon - 2020-03-30

Owned by Gerald Manipon
Last updated: Aug 17, 2020 by Topher Allen
4 min read

Confidence Level High  This article been formally reviewed and is signed off on by a relevant subject matter expert. 

Confidence Level High  This article been formally reviewed and is signed off on by a relevant subject matter expert. 

 

Agenda

  • develop -> develop-es1

  • run end-to-end smoke test to test develop-es1

  • create release v3.0.1 from develop-es1

  • run end-to-end smoke test to test v3.0.1

  • develop-es7 -> develop

  • run end-to-end smoke test to test new develop

  • create release v4.0.0-beta.0 from develop

  • run end-to-end smoke test to test v4.0.0-beta.0

develop -> develop-es1

  • hysds-dockerfiles

    • create develop-es1 branch from develop and update CircleCI config

    • follow what we did for develop-es7

    • once these modifications have been made, the develop-es1 branch of hysds-dockerfiles will build the docker-es1 branches of the puppet modules

    • next we need to create docker-es1 branches for the pertinent puppet modules

  • puppet modules

  • hysds-framework

  • create develop-es1 branches for HySDS core repos (see “All repos updated” section below)

  • push CircleCI build to build the docker images for develop-es1 and deploy them up to dockerhub

    • Since CircleCI is having issues this morning, we will be reusing the hysds/verdi:develop docker image, renaming it to hysds/verdi:develop-es1, and uploading it as a release asset so that the end-to-end smoke test downloads this docker image

Run end-to-end smoke test to test develop-es1

  • Push nisar-pcm jenkins job that tests end-to-end functionality

    • end-to-end succeeded

      null_resource.mozart (remote-exec): + cat /tmp/datasets.txt null_resource.mozart (remote-exec): SUCCESS: Found 66 expected v1.0,2.0,d0001,1 datasets. null_resource.mozart (remote-exec): ++ grep -c '^SUCCESS' /tmp/datasets.txt null_resource.mozart (remote-exec): + SUCCESS=1 null_resource.mozart (remote-exec): + [[ 1 -eq 0 ]] null_resource.mozart (remote-exec): + cat /tmp/check_stamped_dataset_result.txt null_resource.mozart (remote-exec): SUCCESS: Found daac_delivery_status in datasets ['hello_world-product-20200330T191839.415236163Z-0d8e3', 'hello_world-product-20200330T191556.097105247Z-62d17']. null_resource.mozart (remote-exec): ++ grep -c '^SUCCESS' /tmp/check_stamped_dataset_result.txt null_resource.mozart (remote-exec): + SUCCESS=1 null_resource.mozart (remote-exec): + [[ 1 -eq 0 ]] null_resource.mozart (remote-exec): + cat /tmp/cop_catalog.txt null_resource.mozart (remote-exec): SUCCESS: Found 229 expected number of documents in the cop_catalog index null_resource.mozart (remote-exec): ++ grep -c '^SUCCESS' /tmp/cop_catalog.txt null_resource.mozart (remote-exec): + SUCCESS=1 null_resource.mozart (remote-exec): + [[ 1 -eq 0 ]] null_resource.mozart (remote-exec): + cat /tmp/rost_catalog.txt null_resource.mozart (remote-exec): SUCCESS: Found 4 expected number of documents in the rost_catalog index null_resource.mozart (remote-exec): ++ grep -c '^SUCCESS' /tmp/rost_catalog.txt null_resource.mozart (remote-exec): + SUCCESS=1 null_resource.mozart (remote-exec): + [[ 1 -eq 0 ]] null_resource.mozart (remote-exec): + cat /tmp/check_empty_isl_result.txt null_resource.mozart (remote-exec): SUCCESS: No files exist in the ISL Bucket: nisar-dev-isl-fwd-gmanipon++ grep -c '^SUCCESS' /tmp/check_empty_isl_result.txt null_resource.mozart (remote-exec): + SUCCESS=1

create release v3.0.1 from develop-es1

run end-to-end smoke test to test v3.0.1

  • https://nisar-pcm-ci.jpl.nasa.gov/job/force-branches/job/gmanipon-force_branch-E2E-test/24/console

    null_resource.mozart (remote-exec): SUCCESS: Found 66 expected v1.0,2.0,d0001,1 datasets. null_resource.mozart (remote-exec): ++ grep -c '^SUCCESS' /tmp/datasets.txt null_resource.mozart (remote-exec): + SUCCESS=1 null_resource.mozart (remote-exec): + [[ 1 -eq 0 ]] null_resource.mozart (remote-exec): + cat /tmp/check_stamped_dataset_result.txt null_resource.mozart (remote-exec): SUCCESS: Found daac_delivery_status in datasets ['hello_world-product-20200330T233623.680399832Z-33ff7', 'hello_world-product-20200330T233854.951246808Z-64c08']. null_resource.mozart (remote-exec): ++ grep -c '^SUCCESS' /tmp/check_stamped_dataset_result.txt null_resource.mozart (remote-exec): + SUCCESS=1 null_resource.mozart (remote-exec): + [[ 1 -eq 0 ]] null_resource.mozart (remote-exec): + cat /tmp/cop_catalog.txt null_resource.mozart (remote-exec): SUCCESS: Found 229 expected number of documents in the cop_catalog index null_resource.mozart (remote-exec): ++ grep -c '^SUCCESS' /tmp/cop_catalog.txt null_resource.mozart (remote-exec): + SUCCESS=1 null_resource.mozart (remote-exec): + [[ 1 -eq 0 ]] null_resource.mozart (remote-exec): + cat /tmp/rost_catalog.txt null_resource.mozart (remote-exec): SUCCESS: Found 4 expected number of documents in the rost_catalog index null_resource.mozart (remote-exec): ++ grep -c '^SUCCESS' /tmp/rost_catalog.txt null_resource.mozart (remote-exec): + SUCCESS=1 null_resource.mozart (remote-exec): + [[ 1 -eq 0 ]] null_resource.mozart (remote-exec): + cat /tmp/check_empty_isl_result.txt null_resource.mozart (remote-exec): SUCCESS: No files exist in the ISL Bucket: nisar-dev-isl-fwd-gmanipon++ grep -c '^SUCCESS' /tmp/check_empty_isl_result.txt null_resource.mozart (remote-exec): + SUCCESS=1

  • update nightly jobs for hysds_release v3.0.0 → v3.0.1

    • https://nisar-pcm-ci.jpl.nasa.gov/job/nightlies/job/E2E-nisar-pcm-develop-baseline-pge-hysds-release/

    • https://nisar-pcm-ci.jpl.nasa.gov/job/nightlies/job/E2E-nisar-pcm-develop-hysds-release/

    • https://nisar-pcm-ci.jpl.nasa.gov/job/nightlies/job/E2E-nisar-pcm-develop-real-pge/

develop-es7 -> develop

  • puppet modules

    • create annotated tags to bookmark the last ES1.7 commit:

      git checkout develop-es1 git tag -am "Last ES1.7 commit." es1 git checkout docker-es1 git tag -am "Last ES1.7 commit." es1-docker git push --tags

    • update es7 branches so that explicit clones of develop-es7 and docker-es7 are reverted back to develop and docker, respectively.

    • merge develop-es7 into develop:

    • Then:

    • Finally pull develop into master:

All Repos updated

Related Articles:

Have Questions? Ask a HySDS Developer:

Anyone can join our public Slack channel to learn more about HySDS. JPL employees can join #HySDS-Community

JPLers can also ask HySDS questions at Stack Overflow Enterprise

Search HySDS Wiki

Page Information:

Was this page useful?

Yes No

Contribution History:

Subject Matter Expert:

@Gerald Manipon

Find an Error?

Is this document outdated or inaccurate? Please contact the assigned Page Maintainer:

@Gerald Manipon

Note: JPL employees can also get answers to HySDS questions at Stack Overflow Enterprise:

https://stackoverflow.jpl.nasa.gov/