v3.0.0-rc.5

Release Notes for HySDS framework version 3.0.0-rc.5 (LeBron James)

Github release: https://github.com/hysds/hysds-framework/releases/tag/v3.0.0-rc.5

Summary

HySDS framework (core) v3.0.0 release candidate 5 includes bug fixes and major enhancements.

Highlights

• Port from Python 2.7 to Python 3.7+

  • Migration from using OS's built-in python distribution to an Anaconda-based python distribution

• Docker image loading enhancement using Docker registry and S3 backend

• Resiliency improvements when running at high load (NISAR Death Valley workload @ 3000 concurrent workers)

  • Optimizing System Resources and HySDS for Death Valley conditions

• Support for verdi worker execution of GPU job types

• Improvements to job-spec and hysds-io specifications

  • JSON schema validation

  • job-spec fields now required: disk_usage, soft_time_limit, time_limit

• Migration to AWS autoscaling fleet to support multiple instance types and purchase options

• Validation of support for Matlab-based job types

• Various bug fixes in triage post-processor

• Speedup verdi, pge-base, and cuda-pge-base docker images' entrypoint when executing on overlay2 docker storage backend 

All updates

Epic

• HC-10 Closed Port all of hysds core to Python 3

New Feature

• HC-83 Closed decouple job state transitions and other ES queries from event handlers in process_events

• HC-70 Closed Update to optionally skip publishing the built docker image

• HC-55 Closed support to sdscli for AWS autoscaling fleet

• HC-53 Closed add job-spec param for defining required/optional resource attachments for runtime

• HC-51 Closed job-spec and hysds-io JSON validation

• HC-50 Closed make disk_usage, soft_time_limit, and time_limit required parameters in job-spec

• HC-26 Closed Use docker registry for storing container images in s3

• HC-108 Closed update HySDS docker images' entrypoint scripts for faster startup

Bug

• HC-122 Closed enable_dedup is not working for posting job to mozart api

• HC-121 Closed Triage expects all '_' entities to be file which is incorrect

• HC-106 Closed update triage post-processor to create unique triage datasets by default

• HC-77 Closed TypeError when downloading binary file via HTTP

• HC-76 Closed Osaka HTTP Storage isComposite fails when given URL to binary file

• HC-45 Closed Sciflo sets log format with custom attribute on root log handler

• HC-40 Closed Error when triage glob patterns result in files/directories that overlap

• HC-23 Closed Dataset index only gets one alias from a list of aliases

• HC-9 Closed Typo in getting region in build_aws.sh in packer templates

Task

• HC-127 Closed add verdi support for docker usage of GPUs

• HC-107 Closed improve offline job detection in watchdogs

• HC-104 Closed Update ingestion lambda to support regex to job type mapping

• HC-103 Closed Update create cloud staging area command to support non-signal file based triggers

• HC-95 Closed create Matlab-based hello world PGE

• HC-92 Closed update process_events.py to emit celery tasks instead of actually updating ES directly

• HC-91 Closed add event celery workers to Mozart that receives and processes events enqueued from new process_events.py

• HC-78 Closed Command line support for adding queues and instance types for autoscaling fleet

• HC-56 Closed Backport fix for redis.exceptions.DataError: Invalid input of type: 'list'

• HC-48 Closed update redis API call in process_events and set supervisor configuration for autorestart

• HC-47 Closed Dynamic resolution of dependency images in job specs

• HC-44 Closed Expose job retry count in Mozart/figaro

• HC-34 Closed add checksum validator pre-processor

• HC-21 Closed $reset_docker command fails in docker-ephemeral-lvm.sh

• HC-16 Closed Update SSL Configurations

• HC-15 Closed Add support for running on Kubernetes

• HC-5 Closed fix center coordinates determination when dataset polygon intersects the antimerdian

• HC-4 Closed Add publish context capability to verdi

• HC-1 Closed move mozart and grq REST API configuration to jenkins job configuration via sdscli