Version: 1.7 → 7.1

Relevant Github Repos/Branches

...

:

• hysds https://github.com/hysds/hysds.git
• container-builder https://github.com/hysds/container-builder.git
• Rest APIs
  • grq2 https://github.com/hysds/grq2.git
  • mozart https://github.com/hysds/mozart.git
• Good for now, may move to hysds repo
  • hysds_commons https://github.com/hysds/hysds_commons.git
• Will remove because code and scripts ahve been moved to grq/mozart repo
  • figaro https://github.com/hysds/figaro.git
  • tosca https://github.com/hysds/tosca.git
• sdscli https://github.com/sdskit/sdscli.git
• lightweight-jobs https://github.com/hysds/lightweight-jobs.git
• Other repos that are installed in every hysds component
• prov_es https://github.com/hysds/prov_es.git
• chimera https://github.com/hysds/chimera.git
pele https://github.com/hysds/pele.gitRemoval of Tosca/Figaro user interfaces in favor of hysds_ui

Big Changes

• PLEASE LOOK AT AND USE THE NEW ELASTICSEARCH UTILITY CLASS: (SOURCE CODE)

• Only 1 type allowed in each index: _doc
• Need to manually enable all field text searches
• Removal of filtered since ES 5.0
  • documentation
• Split string into text and keyword
  • text allows for more searching capabilities documentation
  • keyword allows for aggregation, etc. Documentation
• fielddata: true is the mapping allows for sorting (but we'll sort on the keyword instead): Documentation
• Support for z coordinate in geoshapes: documentation
  • it wont affect searches but adds more flexibility in location data
• _default_ mapping deprecated in ES 6.0.0 (Link)
  • workaround is using index templates: (Documentation)

• Changes in the geo coordinates query

  • Note: {"type": "geo_shape","tree": "quadtree","tree_levels": 26} makes uploading documents slow, specifically "tree_levels”: 2

  • 
    

    Code Block
    { "query": { "bool": { "filter": { "geo_shape": { "location": { "shape": { "type": "polygon", "coordinates": [[<coordinates>]] }, "relation": "within" } } } } } }

    
    

• Changes in Percolator

  
  
  • Removal of .percolator type Documentation
    • Instead a percolator field type must be configured prior to indexing percolator queries
  • Complete overhaul in the percolate index mapping
    • New version documentation VS. Old version documentation

• Removal of   _all: { "enabled": true }   type in indices so we cannot search for all fields

  • workaround is adding copy_to in field mapping, especially in dynamic templating

  • Link to copy_to documentation

  • Does not work with multi-fields

    • 
      

      Code Block
      "random_field_name": { "type": "keyword", "ignore_above": 256, "copy_to": "all_text_fields", # DOES WORK "fields": { "keyword": { "type": "text" "copy_to": "all_text_fields" # DOES NOT WORK } } }

      
      

  • Proper mapping with text fields

    • 
      

      Code Block
      "random_field_name": { "type": "text", "copy_to": "all_text_fields" "fields": { "keyword": { # WE USE 'raw' instead of 'keyword' in our own indices "type": "keyword" # THIS IS NEEDED FOR AGGREGATION ON THE FACETS FOR THE UI "ignore_above": 256 } } }

      
      

  • Need to add the copy_to field mapping

    • 
      

      Code Block
      "all_text_fields": { "type": "text" }

      
      

• General changes to the mapping

  • create example mapping called grq_v1.1_s1-iw_slc

  • copied example data into new ES index, using built in dynamic mapping to build initial mapping

  • mapping changes:

    • metadata.context to {"type": "object", "enabled": false}

      • properties.location to {"type": "geo_shape","tree": "quadtree"}

      • use type keyword to be able to use msearch:

        • 
          

          Code Block
          "reason": "Fielddata is disabled on text fields by default. ... Alternatively use a keyword field instead."

          
          

• Changes to query_string

  • removal of escaping literal double quotes in query_string
  • old query_string from 1.7, would return S1B_IW_SLC__1SDV_20170812T010949_20170812T011016_006900_00C25E_B16D

    • 
      

      Code Block
      { "query": { "query_string": { "query": "\"__1SDV_20170812T010949_20170812T011016_006900_00C25E_B16\"", "default_operator": "OR" } } }

      
      

  • new query_string returns equivalent document, requires wildcard * at the beginning and end of string

    • 
      

      Code Block
      { "query": { "query_string": { "default_field": "all_text_fields", "query": "*__1SDV_20170812T010949_20170812T011016_006900_00C25E_B16*", "default_operator": "OR" } } }

      
      

  • i dont think date searches really changed much

    • 
      

      Code Block
      { "query": { "query_string": { "query": "starttime: [2019-01-01 TO 2019-01-31]", "default_operator": "OR" } } }

      
      

  • can combine different fields as well

    • 
      

      Code Block
      { "query": { "query_string": { "fields": ["all_text_fields", "all_date_fields"], "query": "[2019-01-01 TO 2019-01-31] AND *__1SDV_20190109T020750_20190109T020817_014411*", "default_operator": "OR" } } }

      
      

...