How to Author a New Dataset Type

Owned by Kate Sammons
Last updated: Jan 20, 2022 by Christopher Rivas
7 min read

Confidence Level TBD  This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it.

Confidence Level TBD  This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it.

 

Dataset Specification:

This page details how to author a new dataset type. In order to for HySDS to recognize a dataset, the dataset must follow certain conventions. These conventions documented on this page must be implemented by the PGE or the PGE wrapper. The dataset conventions include:

  1. Dataset ID:

Each product should have a Dataset ID. This name is used to determine the type of the dataset and name all the important files for the dataset. A Dataset ID is matched against entries found in the datasets.json file to determine its type.

In this example, we shall use the Dataset ID dumby-product-20170101T000000Z-3lx0a.

2. Directory:

Any directory containing the below JSON files and found within the working directory supplied to the PGE is considered a dataset. Thus this directory must be named with the dataset's ID (see above):

$ pwd /data/work/example_work_dir/dumby-product-20170101T000000Z-3lx0a $ ls dumby-product-20170101T000000Z-3lx0a.dataset.json dumby-product-20170101T000000Z-3lx0a.met.json dumby-product-20170101T000000Z-3lx0a.prov_es.json dumby-product-20170101T000000Z-3lx0a.h5 pge_output_2.h5 errors.txt other_metadata.xml



Note: Any other PGE data files should be placed in the Dataset ID directory, as the whole directory is the dataset.



HySDS Dataset and Metadata JSON Files:

1. Dataset JSON file

A product must produce a Dataset ID dataset.json in the Dataset ID directory. This file contains JSON formatted metadata representing the cataloged dataset metadata:

$ cat dumby-product-20170101T000000Z-3lx0a.dataset.json { "version": "v1.0", "label": "dumby product for 2017-01-01T00:00:00Z", "location": { "type": "polygon", "coordinates": [ [ [-122.9059682940358,40.47090915967475], [-121.6679748715316,37.84406528996276], [-120.7310161872557,38.28728069813177], [-121.7043611684245,39.94137004454238], [-121.9536916840953,40.67097860759095], [-122.3100379696548,40.7267890636145], [-122.7640648263371,40.5457010812299], [-122.9059682940358,40.47090915967475] ] ] }, "starttime": "2017-01-01T00:00:00", "endtime": "2017-01-01T00:05:00" }

The required fields are:

*version

The optional fields are:

*label

*location   (in GeoJSON format)

*starttime

*endtime

2. Metadata JSON file

In addition, other metadata data can be added to a Dataset ID met.json in the Dataset ID directory. As long as the file conforms to the JSON format, the dataset developer has free reign on what goes into this file:

$ cat dumby-product-20170101T000000Z-3lx0a.met.json { "startingRange": 800026.4431219272, "sensor": "SAR-C Sentinel1", "esd_threshold": 0.85, "tiles": true, "reference": true, "trackNumber": 144, "lookDirection": "right", "beamMode": "IW", "direction": "descending", "inputFile": "sentinel.ini", "polarization": "VV", "imageCorners": { "maxLon": -117.56055555555555, "minLon": -119.06166666666667, "minLat"

 3. PROV-ES JSON file (optional)

If the developer has instrumented the PGE to use the PROV-ES (Provenence for Earth Science) service, a <Dataset ID>.prov_es.json file must exist in the <Dataset ID> directory containing the PROV-ES JSON:

Dataset Configuration:

This section details how to configure HySDS to recognize a dataset for ingest. Once a PGE is configured to generate a dataset according to the HySDS dataset specification above, users will have to configure HySDS to recognize the new dataset type so that they get ingested properly into the system.

Datasets JSON config

The datasets.json configuration file is centrally stored at some location (usually on the mozart instance in the ops user's ~/.sds/files directory) and is published to a repository where instantiating workers can pull from. At job execution time, the datasets.json configuration file is placed in the work directory of the job for the dataset recognition step. The format for datasets.json is:



At its core the file is a list of data set objects associated with the key "datasets". Each data set object specifies the following properties:

  • ipath: a namespaced name for the product type

  • version: a version string for the product

  • level: a level (L0, L1, L2, L3 ...) string for the product

  • type: type name for the data product

  • id_template: a template to build the id for the product from capture groups from in the match_pattern (#6)

  • match_pattern: regular expression to match product name strings (from sub-directories in PGE working directory). Should always start with a '/'.

  • alt_match_pattern: alternate pattern to #6 (usually just null

  • extractor: path to executable binary or script to run further metadata extraction on the dataset

  • publish: an object specifying how to publish this product composed of the following fields

    • s3-profile-name: AWS profile to use as defined in~/.aws.

    • location: Location for data-store location of final archived product using Osaka prefix

    • urls: List of urls pointing to the product once archived. Can include http access, s3 access or any other URL type

  • browse: an object with the same attributes as publish (#9) but handling the publishing of browse-images



For example, the datasets.json file below shows the configuration for our dumby dataset (dumby-product-20170101T000000Z-3lx0a):







End of formatting revisions.

Start of unrevised content:





= Dataset Specification (how to author a new dataset type) =

In order to for HySDS to recognize a dataset, the dataset must follow certain conventions. These conventions are documented on this page and must be implemented by the PGE or the PGE wrapper.

== Dataset ID ==

Each product should have a dataset ID. This name is used to determine the type of the dataset and name all the important files for the dataset. A dataset ID is matched against entries found in the <code>datasets.json</code> file to determine its type.<br />
In this example, we shall use the dataset ID <code>dumby-product-20170101T000000Z-3lx0a</code>.

== Directory ==

Any directory containing the below JSON files and found within the working directory supplied to the PGE is considered a dataset. Thus this directory must be named with the dataset's ID (see above):

<pre>$ pwd
/data/work/example_work_dir/dumby-product-20170101T000000Z-3lx0a
$ ls
dumby-product-20170101T000000Z-3lx0a.dataset.json
dumby-product-20170101T000000Z-3lx0a.met.json
dumby-product-20170101T000000Z-3lx0a.prov_es.json
dumby-product-20170101T000000Z-3lx0a.h5
pge_output_2.h5
errors.txt
other_metadata.xml
</pre>
''Note that any other PGE data files should be placed in the <Dataset ID> directory, as the whole directory is the dataset.''

== HySDS dataset and metadata JSON files ==

=== dataset JSON file ===

A product must produce a <Dataset ID>.dataset.json in the <Dataset ID> directory. This file contains JSON formatted metadata representing the cataloged dataset metadata:

<pre>$ cat dumby-product-20170101T000000Z-3lx0a.dataset.json
{
"version": "v1.0",
"label": "dumby product for 2017-01-01T00:00:00Z",
"location":

Unknown macro: { "type"}

,
"starttime": "2017-01-01T00:00:00",
"endtime": "2017-01-01T00:05:00"
}
</pre>
The required fields are:

  • <code>version</code>

The optional fields are:

  • <code>label</code>
  • <code>location</code> (in GeoJSON format)
  • <code>starttime</code>
  • <code>endtime</code>

=== metadata JSON file ===

In addition, other metadata data can be added to a <Dataset ID>.met.json in the <Dataset ID> directory. As long as the file conforms to the JSON format, the dataset developer has free reign on what goes into this file:

<pre>$ cat dumby-product-20170101T000000Z-3lx0a.met.json
{
"startingRange": 800026.4431219272,
"sensor": "SAR-C Sentinel1",
"esd_threshold": 0.85,
"tiles": true,
"reference": true,
"trackNumber": 144,
"lookDirection": "right",
"beamMode": "IW",
"direction": "descending",
"inputFile": "sentinel.ini",
"polarization": "VV",
"imageCorners": {
"maxLon": -117.56055555555555,
"minLon": -119.06166666666667,
"minLat"
</pre>

 

 

Have Questions? Ask a HySDS Developer:

Anyone can join our public Slack channel to learn more about HySDS. JPL employees can join #HySDS-Community

JPLers can also ask HySDS questions at Stack Overflow Enterprise

 

Search HySDS Wiki

Page Information:

Page Information:

Was this page useful?

Yes No

Contribution History:

Subject Matter Expert:

@Gerald Manipon

Find an Error?

Is this document outdated or inaccurate? Please contact the assigned Page Maintainer:

@Gerald Manipon



Note: JPL employees can also get answers to HySDS questions at Stack Overflow Enterprise:

https://stackoverflow.jpl.nasa.gov/