v4.0.0

Summary

HySDS framework (core) v4.0.0 includes bug fixes and major enhancements.

Highlights

• migrate from using Elasticsearch 1.7 to Elasticsearch 7.1

  • support using AWS Elasticsearch Service for GRQ backend ES

  • ELK stack 7.1 on metrics

  • major refactoring and unification of Elasticsearch API access from HySDS libraries (hysds, hysds_commons, lightweight-jobs, etc.)

  • Updates to improve ES 7.1 performance

    • remove document bloat in dataset ES JSON payload

    • set default number of ES shards to 8

    • configurable heap size settings

• new HySDS UI replaces figaro and tosca web interfaces

• Chimera framework updates

  • migration of development from internal GHE to github.com

  • add capability to run PGE step inline

• major bug fixes and enhancements to osaka and harikiri

• centralization of user rules for jobs and datasets on mozart ES

• update anaconda python distribution to python3.8

• Major cleanup of deprecated code

• Fix issue in RFC_6598 IP range

• Initial integration of SDSWatch and a minimal OPS UI

  • added SDSWatch loggers for systemd and supervisord services on PCM components

• Various updates and bug fixes to user interfaces

• Remove job deduplication ambiguity

  • jobs must explicitly set enable_dedup to false in their hysds-io.json

• allow override of the default no-clobber behavior of dataset publishing

All updates