OpenSearch Serverless

Owned by Dustin Lo
Last updated: Dec 05, 2022
4 min read

Sources:

Cost

  • OpenSearch serverless needs a "server" running called a OCU (OpenSearch Compute Unit)

    • billed at $0.24/hr, with a minimum of 2 OCUs running at once

    • “Each OCU is a combination of 6 GiB of memory and corresponding virtual CPU (vCPU), as well as data transfer to Amazon S3.”

    • The cost per month of data stored in Amazon S3 is $0.024 per GB

    • no charges for OpenSearch fashboard usage

Supported Regions

  • as of Dec '23, it is supported in these regions

Region Name

Region

Endpoint

Protocol

Region Name

Region

Endpoint

Protocol

US East (Ohio)

us-east-2

aoss.us-east-2.amazonaws.com

HTTPS

US East (N. Virginia)

us-east-1

aoss.us-east-1.amazonaws.com

HTTPS

US West (Oregon)

us-west-2

aoss.us-west-2.amazonaws.com

HTTPS

Asia Pacific (Tokyo)

ap-northeast-1

aoss.ap-northeast-1.amazonaws.com

HTTPS

Europe (Ireland)

eu-west-1

aoss.eu-west-1.amazonaws.com

HTTPS

Benefits

  • Simpler than provisioned – OpenSearch Serverless removes much of the complexity of managing OpenSearch clusters and capacity. It automatically sizes and tunes your clusters, and takes care of shard and index lifecycle management. It also manages service software updates and OpenSearch version upgrades. All updates and upgrades are non-disruptive.

    • Cost-effective – When you use OpenSearch Serverless, you only pay for the resources that you consume. This removes the need for upfront provisioning and overprovisioning for peak workloads.

    • Highly available – OpenSearch Serverless supports production workloads with redundancy to protect against Availability Zone outages and infrastructure failures.

    • Scalable – OpenSearch Serverless automatically scales resources to maintain consistently fast data ingestion rates and query response times.

Logstash

input { s3 { bucket => "my-s3-bucket" region => "us-east-1" } } output { opensearch { ecs_compatibility => disabled hosts => "https://my-collection-endpoint.us-east-1.aoss.amazonaws.com:443" index => my-index auth_type => { type => 'aws_iam' aws_access_key_id => 'your-access-key' aws_secret_access_key => 'your-secret-key' region => 'us-east-1' service_name => 'aoss' } default_server_major_version => 2 legacy_template => false } }

Signing Requests

  • You must specify the service name as aoss.

  • You can't include Content-Length as a signed header, otherwise you'll get an invalid signature error.

  • The x-amz-content-sha256 header is required for all AWS Signature Version 4 requests. It provides a hash of the request payload. For OpenSearch Serverless, include it with one of these values when you build the canonical request for signing:

  • If there's a request payload, set the value to its Secure Hash Algorithm (SHA) cryptographic hash (SHA256).

  • If there's no request payload, set the value to e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, which is the hash of an empty string.

  • In either of the above two cases, you can also use the literal string UNSIGNED-PAYLOAD as the value of the x-amz-content-sha256 header.

API Operations

Not sure if it supports certain endpoints, such as:

  • Point-in-time (PIT) API: POST /<index>/_pit?keep_alive=1m

  • Scroll API: GET /_search/scroll

aoss:CreateIndex

PUT <index>

Create indexes. For more information, see Create index.

aoss:DescribeIndex

  • GET <index>

  • GET <index>/_mapping

  • GET <index>/_mappings

  • GET <index>/_setting

  • GET <index>/_setting/<setting>

  • GET <index>/_settings

  • GET <index>/_settings/<setting>

  • GET _cat/indices

  • GET _mapping

  • GET _mappings

  • GET _resolve/index/<index>

Describe indexes. For more information, see the following resources:

aoss:WriteDocument

  • DELETE <index>/_doc/<id> (for search collection types only)

  • POST <index>/_bulk

  • POST <index>/_create/<id> (for search collection types only)

  • POST <index>/_doc

  • POST <index>/_update/<id> (for search collection types only)

  • POST _bulk

  • PUT <index>/_create/<id> (for search collection types only)

  • PUT <index>/_doc/<id> (for search collection types only)

Write and update documents. For more information, see the following resources:

Note

Some operations are only allowed for collections of type SEARCH. For more information, see Choosing a collection type.

aoss:ReadDocument

  • GET <index>/_analyze

  • GET <index>/_doc/<id>

  • GET <index>/_explain/<id>

  • GET <index>/_mget

  • GET <index>/_source/<id>

  • GET <index>/_count

  • GET <index>/_field_caps

  • GET <index>/_msearch

  • GET <index>/_rank_eval

  • GET <index>/_search

  • GET <index>/_validate/<query>

  • GET _analyze

  • GET _field_caps

  • GET _mget

  • GET _search

  • HEAD <index>/_doc/<id>

  • HEAD <index>/_source/<id>

  • POST <index>/_analyze

  • POST <index>/_explain/<id>

  • POST <index>/_count

  • POST <index>/_field_caps

  • POST <index>/_rank_eval

  • POST <index>/_search

  • POST _analyze

  • POST _field_caps

  • POST _search

Read documents. For more information, see the following resources:

aoss:DeleteIndex

DELETE <target>

Delete indexes. For more information, see Delete index.

aoss:UpdateIndex

  • POST _mapping

  • POST <index>/_mapping/

  • POST <index>/_mappings/

  • POST <index>/_setting

  • POST <index>/_settings

  • POST _setting

  • POST _settings

  • PUT _mapping

  • PUT <index>/_mapping

  • PUT <index>/_mappings/

  • PUT <index>/_setting

  • PUT <index>/_settings

  • PUT _setting

  • PUT _settings

Update index settings. For more information, see the following resources:

aoss:CreateCollectionItems

POST _aliases

Create index aliases. For more information, see Create aliases.

aoss:DescribeCollectionItems

  • GET <index>/_alias/<alias>

  • GET _alias

  • GET _alias/<alias>

  • GET _cat/aliases

  • GET _cat/templates

  • GET _cat/templates/<template_name>

  • GET _component_template

  • GET _component_template/<component-template>

  • GET _index_template

  • GET _index_template/<index-template>

  • HEAD _alias/<alias>

  • HEAD _component_template/<component-template>

  • HEAD _index_template/<name>

  • HEAD <index>/_alias/<alias>

Describe aliases and index templates. For more information, see the following resources:

aoss:UpdateCollectionItems

  • POST <index>/_alias/<alias>

  • POST <index>/_aliases/<alias>

  • POST _component_template/<component-template>

  • POST _index_template/<index-template>

  • PUT <index>/_alias/<alias>

  • PUT <index>/_aliases/<alias>

  • PUT _component_template/<component-template>

  • PUT _index_template/<index-template>

Update aliases and index templates. For more information, see the following resources:

aoss:DeleteCollectionItems

  • DELETE <index>/_alias/<alias>

  • DELETE _component_template/<component-template>

  • DELETE _index_template/<index-template>

  • DELETE <index>/_aliases/<alias>

Delete aliases and index templates. For more information, see the following resources:

Drawbacks

  • Not all OpenSearch API operations supported

  • can't take snapshot or restore snapshots

  • index refresh interval varies between 10-30 seconds, depending on size of requests

    • opposed to the 10 sec default in Elasticsearch

    • might affect user_rule processing

  • OpenSearch Serverless automatically upgrades your collections to new OpenSearch versions. Upgrades don't necessarily happen as soon as a new version is available.

    • this can be potentially bad as it can break HySDS if it doesnt support the latest version

  • Not a mature product, still in a preview mode for users

    • documentation a little incomplete

    • will have to test out features to see if they exist in OpenSearch Serverless

Still needs testing

  • HySDS UI

    • currently we need a reverse proxy for hysds_ui to communicate with AWS ES

      • we use a forked version of aws-es-proxy: https://github.com/hysds/aws-es-proxy

      • will need to make changes to code to accommodate for OpenSearch Serverless

        • have a flag for --service which can be set to aoss (default will be es)

      • Logstash

        • logstash-plugin install --version 0.1.0.pre logstash-integration-aws logstash-plugin install --version 2.0.0 logstash-output-opensearch

  • will need to test if the existing elasticsearch-python (>=7.0.0,<7.14.0) will work with OpenSearch Serverless

  • Deep pagination:

    • Starting from ElasticSearch 7.0, the limit of records returned by the _search API is 10,000 records

      • This led to us relying on the scroll API for deep pagination, and eventually the PIT + search_after API

        • not sure if that still applies to OpenSearch Serverless, or if they do not impose a hard limit on the _search API

      • a lot of HySDS core + PCM code uses it

 

 

Note: JPL employees can also get answers to HySDS questions at Stack Overflow Enterprise:

https://stackoverflow.jpl.nasa.gov/