Useful links:

• https://www.elastic.co/blog/whats-new-elasticsearch-7-10-0-searchable-snapshots-store-more-for-less

• https://www.elastic.co/guide/en/elasticsearch/reference/7.10/breaking-changes-7.0.html

• https://docs.aws.amazon.com/opensearch-service/latest/developerguide/what-is.html

Changes in Elasticsearch 7.10

• benchmarks have reported space reductions of up to 10% using a new stored field compression

• search.max_buckets now defaults to 10,000

  • will severely limit and affect aggregation queries in PCM code

  • we will have to switch to composite aggregation beforehand (NSDS-2245)

• Cluster-wide shard soft limit

  • Clusters now have soft limits on the total number of open shards in the cluster based on the number of nodes and the cluster.max_shards_per_node cluster setting

  • may need to reduce the # of shards in settings in case we are using a single node cluster

• : is no longer allowed in index name

  • Due to cross-cluster search using : to separate a cluster and index name, index names may no longer contain :

• Deprecated geo_shape parameters

  • The following type parameters are deprecated for the geo_shape field type: tree, precision, tree_levels, distance_error_pct, points_only, and strategy. They will be removed in a future version.

• Point-in-time (PIT) queries:

  • lightweight deep pagination

  • Elasticsearch is moving away from the scroll API

    • no more dealing with 500 active scroll_id's at a time

  • PIT queries gives a view into the state of the data when it was initiated

    • refreshes will not affect the data

  • # must be opened explicitly POST /my-index-000001/_pit?keep_alive=1m # executing the query POST /_search { "size": 100, "query": { "match" : { "title" : "elasticsearch" } }, "pit": { "id": "46ToAwMDaWR4BXV1aWQxAgZub2RlXzEAAAAAAAAAAAEBYQNpZHkFdXVpZDIrBm5vZGVfMwAAAAAAAAAAKgFjA2lkeQV1dWlkMioGbm9kZV8yAAAAAAAAAAAMAWICBXV1aWQyAAAFdXVpZDEAAQltYXRjaF9hbGw_gAAAAA==", "keep_alive": "1m" } } # deleting the PIT query DELETE /_pit { "id" : "46ToAwMDaWR4BXV1aWQxAgZub2RlXzEAAAAAAAAAAAEBYQNpZHkFdXVpZDIrBm5vZGVfMwAAAAAAAAAAKgFjA2lkeQV1dWlkMioGbm9kZV8yAAAAAAAAAAAMAWIBBXV1aWQyAAA=" }

• this is Elasticsearch's future plan for deep pagination so we should change the query method in hysds_common's ElasticsearchUtility class
  - if version < 7.10, use scroll API, if version >= 7.10, use search after API
  - this will allow for backwards compatibility with HySDS + different versions of ES

Level-of-effort:

• (MEDIUM) creating "snapshot" of ec2 instance with elasticsearch 7.10 + geonames index

  • we have a snapshot of geonames in s3 saved so restoring it will be fast and straight forward

  • no need to populate the geonames index manually, can take 2+ days

  • coordinate with Susan to get this done

• (MEDIUM, OPTIONAL) changes to hysds_common's query method:

  • if version < 7.10, use scroll API

  • if version >= 7.10, use point-in-time API
    - https://packaging.pypa.io/en/latest/version.html#packaging.version.parse

    from packaging import version version.parse("7.9.3") >= version.parse("7.10") # False

• (LOW) sanity testing NISAR & SWOT with Elasticsearch 7.10

  • NISAR: dev-e2e forward & reprocessing, real-pge, create & restore snapshot

  • SWOT: need consult from @Michael Cayanan

• (LOW) fix NISAR e2e test with AWS Elasticsearch