Page Comparison

Release Notes for HySDS framework version 3.0.0-rc.5 (LeBron James)

Github release: https://github.com/hysds/hysds-framework/releases/tag/v3.0.0-rc.5

Summary

HySDS framework (core) v3.0.0 release candidate 5 includes bug fixes and major enhancements.

Highlights

• Port from Python 2.7 to Python 3.7+

  • Migration from using OS's built-in python distribution to an Anaconda-based python distribution

• Docker image loading enhancement using Docker registry and S3 backend

• Resiliency improvements when running at high load (NISAR Death Valley workload @ 3000 concurrent workers)

  • Optimizing System Resources and HySDS for Death Valley conditions

• Support for verdi worker execution of GPU job types

• Improvements to job-spec and hysds-io specifications

  • JSON schema validation

  • job-spec fields now required: disk_usage, soft_time_limit, time_limit

• Migration to AWS autoscaling fleet to support multiple instance types and purchase options

• Validation of support for Matlab-based job types

• Various bug fixes in triage post-processor

• Speedup verdi, pge-base, and cuda-pge-base docker images' entrypoint when executing on overlay2 docker storage backend 

All updates

Epic

• HC-10

  Status
  subtle true colour

• Green title Closed

  Port all of hysds core to Python 3

New Feature

• HC-83

  Status
  subtle true colour

• Green title Closed

  decouple job state transitions and other ES queries from event handlers in process_events

• HC-70

  Status
  subtle true colour

• Green title Closed

  Update to optionally skip publishing the built docker image

• HC-55

  Status
  subtle true colour

• Green title Closed

  support to sdscli for AWS autoscaling fleet

• HC-53

  Status
  subtle true colour

• Green title Closed

  add job-spec param for defining required/optional resource attachments for runtime

• HC-51

  Status
  subtle true colour

• Green title Closed

  job-spec and hysds-io JSON validation

• HC-50

  Status
  subtle true colour

• Green title Closed

  make disk_usage, soft_time_limit, and time_limit required parameters in job-spec

• HC-26

  Status
  subtle true colour

• Green title Closed

  Use docker registry for storing container images in s3

• HC-108

  Status
  subtle true colour

• Green title Closed

  update HySDS docker images' entrypoint scripts for faster startup

Bug

• HC-122

  Status
  subtle true colour

• Green title Closed

  enable_dedup is not working for posting job to mozart api

• HC-121

  Status
  subtle true colour

• Green title Closed

  Triage expects all '_' entities to be file which is incorrect

• HC-106

  Status
  subtle true colour

• Green title Closed

  update triage post-processor to create unique triage datasets by default

• HC-77

  Status
  subtle true colour

• Green title Closed

  TypeError when downloading binary file via HTTP

• HC-76

  Status
  subtle true colour

• Green title Closed

  Osaka HTTP Storage isComposite fails when given URL to binary file

• HC-45

  Status
  subtle true colour

• Green title Closed

  Sciflo sets log format with custom attribute on root log handler

• HC-40

  Status
  subtle true colour

• Green title Closed

  Error when triage glob patterns result in files/directories that overlap

• HC-23

  Status
  subtle true colour

• Green title Closed

  Dataset index only gets one alias from a list of aliases

• HC-9

  Status
  subtle true colour

• Green title Closed

  Typo in getting region in build_aws.sh in packer templates

Task

• HC-127

  Status
  subtle true colour

• Green title Closed

  add verdi support for docker usage of GPUs

• HC-107

  Status
  subtle true colour

• Green title Closed

  improve offline job detection in watchdogs

• HC-104

  Status
  subtle true colour

• Green title Closed

  Update ingestion lambda to support regex to job type mapping

• HC-103

  Status
  subtle true colour

• Green title Closed

  Update create cloud staging area command to support non-signal file based triggers

• HC-95

  Status
  subtle true colour

• Green title Closed

  create Matlab-based hello world PGE

• HC-92

  Status
  subtle true colour

• Green title Closed

  update process_events.py to emit celery tasks instead of actually updating ES directly

• HC-91

  Status
  subtle true colour

• Green title Closed

  add event celery workers to Mozart that receives and processes events enqueued from new process_events.py

• HC-78

  Status
  subtle true colour

• Green title Closed

  Command line support for adding queues and instance types for autoscaling fleet

• HC-56

  Status
  subtle true colour

• Green title Closed

  Backport fix for redis.exceptions.DataError: Invalid input of type: 'list'

• HC-48

  Status
  subtle true colour

• Green title Closed

  update redis API call in process_events and set supervisor configuration for autorestart

• HC-47

  Status
  subtle true colour

• Green title Closed

  Dynamic resolution of dependency images in job specs

• HC-44

  Status
  subtle true colour

• Green title Closed

  Expose job retry count in Mozart/figaro

• HC-34

  Status
  subtle true colour

• Green title Closed

  add checksum validator pre-processor

• HC-21

  Status
  subtle true colour

• Green title Closed

  $reset_docker command fails in docker-ephemeral-lvm.sh

• HC-16

  Status
  subtle true colour

• Green title Closed

  Update SSL Configurations

• HC-15

  Status
  subtle true colour

• Green title Closed

  Add support for running on Kubernetes

• HC-5

  Status
  subtle true colour

• Green title Closed

  fix center coordinates determination when dataset polygon intersects the antimerdian

• HC-4

  Status
  subtle true colour

• Green title Closed

  Add publish context capability to verdi

• HC-1

  Status
  subtle true colour

• Green title Closed

  move mozart and grq REST API configuration to jenkins job configuration via sdscli