What's new in Python 3.9

Related ticket(s):

• https://hysds-core.atlassian.net/browse/HC-385

• https://jira.jpl.nasa.gov/browse/NSDS-1555

Related repos:

• hysds-dbxml: https://github.com/pymonger/hysds-dbxml

• puppet-hysds_base: https://github.com/hysds/puppet-hysds_base/tree/HC-385

• sciflo:

• puppet-verdi:

• lightweight-jobs:

• nisar-pcm: https://github.jpl.nasa.gov/IEMS-SDS/nisar-pcm/tree/NSDS-1555

• pcm-releaser: https://github.jpl.nasa.gov/IEMS-SDS/pcm-releaser/tree/NSDS-1555

• CNM_product_delivery: https://github.jpl.nasa.gov/IEMS-SDS/CNM_product_delivery/tree/NSDS-1555

if you want to test your PGEs with the hysds/pge-base:python3.9 docker image you can SSH into CI and export the image with docker save and import it with docker load

You can build the docker images yourself if you don’t want to export/import image elsewhere, you would have to run (w/ branch HC-385) in your CI instance

• warning: it can take a while to build the images & there are a lot of steps as well (some missing from here)

• example Circle CI build taking 2+ hrs

will build the images for with Dockerfile, Dockerfile.pge-base, Dockerfile.cuda-pge-base

able to confirm python 3.9 running in all docker images

Make sure all your python dependencies are compatible with 3.9:

• numpy 1.19.2 only supports 3.6-3.8, make sure to fix it to numpy>1.19.2 in your requirements.txt or setup.py to avoid this error:

• Some libraries that are compatible with 3.9:

  • pytest

  • scripttest

  • mock

  • mockito

  • flake8

  • pytest-cov

  • flake8-junit-report

  • flake8-string-format

  • xmltodict

  • yamale==2.0.1

  • ruamel.yaml

  • elasticmock

  • geopandas

  • gdal (it works in NISAR but would have to do some more digging)

  • elasticsearch (use version <7.14.0 b/c of licensing issues)

• if you’re using conda and the cffi library, make sure to update the version to 1.14.6

  • 1.14.0 does not work with python 3.9 (haven't tested 1.14.1, 1.14.2, 1.14.3, 1.14.4, 1.14.5)

Changes to related project repos & Terraform cluster provisioner

• https://github.jpl.nasa.gov/IEMS-SDS/pcm-releaser/compare/NSDS-1555

• https://github.jpl.nasa.gov/IEMS-SDS/nisar-pcm/compare/NSDS-1555

• https://github.jpl.nasa.gov/IEMS-SDS/CNM_product_delivery/tree/NSDS-1555

A kluge was added to both pcm-releaser and puppet-hysds_base to allow the provisioner to install the latest version of miniconda which uses python 3.9

• https://github.jpl.nasa.gov/IEMS-SDS/pcm-releaser/compare/NSDS-1555

Testing

Was able to run a dev-e2e (both fwd and reprocessing for NISAR), the output logs are attached in the NSDS-1555 ticket

• all dataset checks passed

• no jobs failed

docker images built for nisar-pcm and CNM_product_delivery

• after fixing the versions for dependencies

No core functionality broke:

• Rest APIs working (mozart + GRQ2)

• celery workers are functioning

  • orchestrator

  • user rules processor (for datasets and jobs)

• lightweight-jobs was tested with the following jobs working:

  • purge dataset

  • purge job

  • echo

  • revoke job

  • retry job

  • AWS get