Agenda

• develop -> develop-es1

• run end-to-end smoke test to test develop-es1

• create release v3.0.1 from develop-es1

• run end-to-end smoke test to test v3.0.1

• develop-es7 -> develop

• run end-to-end smoke test to test new develop

• create release v4.0.0-beta.0 from develop

• run end-to-end smoke test to test v4.0.0-beta.0

develop -> develop-es1

• hysds-dockerfiles

  • create develop-es1 branch from develop and update CircleCI config

  • follow what we did for develop-es7

  • once these modifications have been made, the develop-es1 branch of hysds-dockerfiles will build the docker-es1 branches of the puppet modules

  • next we need to create docker-es1 branches for the pertinent puppet modules

• puppet modules

  • puppet-verdi, puppet-mozart, puppet-metrics, puppet-grq, puppet-cont_int

    • develop → develop-es1

      • update develop-es1 branch, e.g.: https://github.com/hysds/puppet-verdi/compare/develop...develop-es1

    • docker → docker-es1

      • update docker-es1 branch, e.g.: https://github.com/hysds/puppet-verdi/compare/docker...docker-es1

      • this will install hysds framework based on develop-es1 branch which will build off of develop-es1 branch of core repos

• hysds-framework

  • develop → develop-es1

  • update develop-es1, e.g.: https://github.com/hysds/hysds-framework/compare/develop...develop-es1

• create develop-es1 branches for HySDS core repos (see “All repos updated” section below)

• push CircleCI build to build the docker images for develop-es1 and deploy them up to dockerhub

  • Since CircleCI is having issues this morning, we will be reusing the hysds/verdi:develop docker image, renaming it to hysds/verdi:develop-es1, and uploading it as a release asset so that the end-to-end smoke test downloads this docker image

Run end-to-end smoke test to test develop-es1

• Push nisar-pcm jenkins job that tests end-to-end functionality

  • end-to-end succeeded

    null_resource.mozart (remote-exec): + cat /tmp/datasets.txt null_resource.mozart (remote-exec): SUCCESS: Found 66 expected v1.0,2.0,d0001,1 datasets. null_resource.mozart (remote-exec): ++ grep -c '^SUCCESS' /tmp/datasets.txt null_resource.mozart (remote-exec): + SUCCESS=1 null_resource.mozart (remote-exec): + [[ 1 -eq 0 ]] null_resource.mozart (remote-exec): + cat /tmp/check_stamped_dataset_result.txt null_resource.mozart (remote-exec): SUCCESS: Found daac_delivery_status in datasets ['hello_world-product-20200330T191839.415236163Z-0d8e3', 'hello_world-product-20200330T191556.097105247Z-62d17']. null_resource.mozart (remote-exec): ++ grep -c '^SUCCESS' /tmp/check_stamped_dataset_result.txt null_resource.mozart (remote-exec): + SUCCESS=1 null_resource.mozart (remote-exec): + [[ 1 -eq 0 ]] null_resource.mozart (remote-exec): + cat /tmp/cop_catalog.txt null_resource.mozart (remote-exec): SUCCESS: Found 229 expected number of documents in the cop_catalog index null_resource.mozart (remote-exec): ++ grep -c '^SUCCESS' /tmp/cop_catalog.txt null_resource.mozart (remote-exec): + SUCCESS=1 null_resource.mozart (remote-exec): + [[ 1 -eq 0 ]] null_resource.mozart (remote-exec): + cat /tmp/rost_catalog.txt null_resource.mozart (remote-exec): SUCCESS: Found 4 expected number of documents in the rost_catalog index null_resource.mozart (remote-exec): ++ grep -c '^SUCCESS' /tmp/rost_catalog.txt null_resource.mozart (remote-exec): + SUCCESS=1 null_resource.mozart (remote-exec): + [[ 1 -eq 0 ]] null_resource.mozart (remote-exec): + cat /tmp/check_empty_isl_result.txt null_resource.mozart (remote-exec): SUCCESS: No files exist in the ISL Bucket: nisar-dev-isl-fwd-gmanipon++ grep -c '^SUCCESS' /tmp/check_empty_isl_result.txt null_resource.mozart (remote-exec): + SUCCESS=1

create release v3.0.1 from develop-es1

• https://github.com/hysds/hysds-framework/releases/tag/v3.0.1

• awaiting creation and publishing of v3.0.` verdi docker image on circleci: https://app.circleci.com/pipelines/github/hysds/hysds-framework/102/workflows/84ca3118-9185-410c-b77d-20548e346905/jobs/198

  • published: https://github.com/hysds/hysds-framework/releases/tag/v3.0.1

• if a new release was created for lightweight-jobs, ensure that hysds-verdi-latest.tar.gz and docker-registry-2.tar.gz was published as assets

run end-to-end smoke test to test v3.0.1

• https://nisar-pcm-ci.jpl.nasa.gov/job/force-branches/job/gmanipon-force_branch-E2E-test/24/console

  null_resource.mozart (remote-exec): SUCCESS: Found 66 expected v1.0,2.0,d0001,1 datasets. null_resource.mozart (remote-exec): ++ grep -c '^SUCCESS' /tmp/datasets.txt null_resource.mozart (remote-exec): + SUCCESS=1 null_resource.mozart (remote-exec): + [[ 1 -eq 0 ]] null_resource.mozart (remote-exec): + cat /tmp/check_stamped_dataset_result.txt null_resource.mozart (remote-exec): SUCCESS: Found daac_delivery_status in datasets ['hello_world-product-20200330T233623.680399832Z-33ff7', 'hello_world-product-20200330T233854.951246808Z-64c08']. null_resource.mozart (remote-exec): ++ grep -c '^SUCCESS' /tmp/check_stamped_dataset_result.txt null_resource.mozart (remote-exec): + SUCCESS=1 null_resource.mozart (remote-exec): + [[ 1 -eq 0 ]] null_resource.mozart (remote-exec): + cat /tmp/cop_catalog.txt null_resource.mozart (remote-exec): SUCCESS: Found 229 expected number of documents in the cop_catalog index null_resource.mozart (remote-exec): ++ grep -c '^SUCCESS' /tmp/cop_catalog.txt null_resource.mozart (remote-exec): + SUCCESS=1 null_resource.mozart (remote-exec): + [[ 1 -eq 0 ]] null_resource.mozart (remote-exec): + cat /tmp/rost_catalog.txt null_resource.mozart (remote-exec): SUCCESS: Found 4 expected number of documents in the rost_catalog index null_resource.mozart (remote-exec): ++ grep -c '^SUCCESS' /tmp/rost_catalog.txt null_resource.mozart (remote-exec): + SUCCESS=1 null_resource.mozart (remote-exec): + [[ 1 -eq 0 ]] null_resource.mozart (remote-exec): + cat /tmp/check_empty_isl_result.txt null_resource.mozart (remote-exec): SUCCESS: No files exist in the ISL Bucket: nisar-dev-isl-fwd-gmanipon++ grep -c '^SUCCESS' /tmp/check_empty_isl_result.txt null_resource.mozart (remote-exec): + SUCCESS=1

• update nightly jobs for hysds_release v3.0.0 → v3.0.1

  • https://nisar-pcm-ci.jpl.nasa.gov/job/nightlies/job/E2E-nisar-pcm-develop-baseline-pge-hysds-release/

  • https://nisar-pcm-ci.jpl.nasa.gov/job/nightlies/job/E2E-nisar-pcm-develop-hysds-release/

  • https://nisar-pcm-ci.jpl.nasa.gov/job/nightlies/job/E2E-nisar-pcm-develop-real-pge/

develop-es7 -> develop

• puppet modules

  • create annotated tags to bookmark the last ES1.7 commit:

    git checkout develop-es1 git tag -am "Last ES1.7 commit." es1 git checkout docker-es1 git tag -am "Last ES1.7 commit." es1-docker git push --tags

  • update es7 branches so that explicit clones of develop-es7 and docker-es7 are reverted back to develop and docker, respectively.

  • merge develop-es7 into develop:

  • Then:

  • Finally pull develop into master:

All Repos updated

• hysds https://github.com/hysds/hysds.git

  • PR: https://github.com/hysds/hysds/pull/52

• hysds-framework

  • PR: https://github.com/hysds/hysds-framework/pull/15

• container-builder 

  • PR:

• Rest APIs

  • grq2 

    • PR:

  • mozart 

    • PR:

• Good for now, may move to hysds repo

  • hysds_commons 

• Will remove because code and scripts ahve been moved to grq/mozart repo

  • figaro 

  • tosca 

    • PR:

• sdscli 

  • PR:

• lightweight-jobs 

  • PR:

• Other repos that are installed in every hysds component

  • prov_es 

  • chimera 

    • PR:

  • pele 

    • PR: