Introduction

HySDS has a number of APIs that are designed to provide a way for external users to programmatically access HySDS data and job information.  The APIs are also meant to throttle or jitter requests to avoid hammering the backend servers and datastores.

As we evolve HySDS, the most important role that APIs may play is to throttle or jitter the requests from HySDS itself, which may hammer our backend services. 

This is an initial attempt to compile information regarding the existing APIs and come upon best practices and design for new APIs or evolution of existing APIs. 

• We tried to build on top of Swagger UI

Going forward

Even within the SDS, we find a need to have an API or middleware layer to communicate between components, to avoid having every developer come up with their own way to talk to components or the underlying services.

Longterm reward to investing in API now

• extracting best practices from existing code that can be reused in a generic way

• Then refactoring existing code to call API instead of using custom code

Questions

• how do you separate traffic from external users from internal SDS machinery?

• All internal SDS machinery calls would be done as the special "ops" user.

• In HySDS core -- there is the "ops" super-user.  Everyone else has limited access, based off explicitly granting them access.

• how do we get requirements from Ops and science users.

• Should everything be a REST API?

  • We use boto python library to talk to Amazon

• Where do we go with SDS Watch

  * should be a product.  Should interface via API.  May need to interact with Amazon.

Suggestion:

• abstract the backend --

  • Uniform way of dealing with common issues and scenarios

  • jitter and rate limits to avoid hammering backend service

  • common place to deal with anomalous service response in consistent manne

  •  

    • distinguish from "no results returned" and "service unavailable"

    • automatic retry with exponential backoff

  • we can write out JSON query...but parsing ElasticSearch results should be abstracted and handled by API.

  • We can catch bad ES responses via API

  • We could sanitize JSON query against bad requests, but otherwise let the query through.

•  

  • will make it easier for upgrading or swapping backend services.

  • encapsulate best practices in a common place

  • No elasticsearch code entangled with HySDS and PGE codebase

  • Makes it easier for end-users to develop tools and for development of PGEs.

  • ES abstraction layer should have the following characteristics (build upon query-util?)

    • (1) jittering, with automated & finite exponential backoff to backend (ES)
      (2) rate limits on API for throttling to backend (ES)
      (3) reuse connections to backend (ES) to improve performance 

• for public APIs

  • register with email and password

  •  

    • help identify our users for announcements, or directed emails.

    • results depend on user access control, i.e. what should be visible to them.

  • programmatic way to refresh token

  •  

    • require authentication/token

    • require time limits on tokens.

  • assign limits per user

  •  

    • ability to throttle or limit bad end-users, so they don't cause denial of service to others.

  • versioning of APIs to have path to updated APIs, deprecating old verisons

  • collect metrics regarding API use

  •  

    • versioning

    • which actions are called the most

    • performance analysis

  • helpful for developers to debug. 

  • Also helpful to provide documentation to end user.

•  

  • additional requirements

  • swagger-ui

Pele Links and Examples

Code can be found in hysds/pele

Pele - enables us to query datasets -- clean up the terminology. -- Gerald may be documenting this in SDD

Was developed mainly for Osiris (urgent response UI), based on what Namrata needed during development.

• Currently, main user is Osiris (urgent response UI)

• Future users -- science users who want to pull data programmatically, ops reporting

Functionality

• dataset types

• datasets

• list datasets of a certain type.  (are results paginated?)

• get ids by dataset (so we can pull out more specific info later)

• get ids by type

• get dataset by id

• query for certain fields for specified type and dataset.

• list datasets that overlap temporally (by ID) or spatially

•  

  • Used to grab all acquisitions or product types that match AOI spatially and temporal extent

• list overlapping datasets of particular types

•  

  • check for each type whether products exist or not

  • for example, only care about COD or LAR or SLCP

•  

  • swagger api requires authentication

  • need to authenticate for post, but not gets (confirm with Gerald)

  • dataset and dataset type.

  • get datasettype based on dataset

Mozart Links and Examples

• in lambdas on AWS that trigger job submissions -- mostly for ASF delivery.  Will be using it for scheduled lambdas.  (Instead of cronjob)

• ASF has put jobs into our system

• FEMA could've put processing jobs in our system

• CI to register jobs (including job spec) into Mozart + GRQ

• MyJobs????

Functionality

• list queues

• manage job specs -- add, list, remove, get job type ID.  (given job type, returns the actual job spec as JSON)

• manage jobs --- get info by ID, list submitted jobs, get status based on job id, submit a job

  • Namrata could name the job, but HySDS will append a timestamp to that name.   (This is more common inside HySDS machinery when using hysds utils)

•  

  • Wish for enhancement to allow job naming when submitting through REST API

  • Wish to query jobs by user

  • Wish to list all known users

• manage containers -- used mainly by CI

• hysds_io - manage only Mozart actions

• event - gives system ability to publish events to log anomalies like spot termination, etc.

GRQ Links and Examples

• used by CI

Functionality

• register jobs and actions to GRQ

SMAP-API Links and Examples

SMAP-API

• GitHub-FN

Functionality

•  

  • get product by id

  • get job by id

  • get docs in an index

  • list all half orbit statuses using specific metadata

  • list all half orbits