Product Metadata Files

This article provides a brief overview of the .JSON log files found in each completed science product produced by HySDS. These files are located at the root directory of the worker node.

The Elasticsearch database is built by combining info from _context.json, _met.json, and _dataset.json. In a working directory this is among the bare minimum required for HySDS to recognize it as a valid dataset. There may be additional information, but this is the essentials.

The three main log files are:

_context.json

This file provides a unique “recipe” for each job. Once a job is picked up by a Verdi worker, the _context.json file is updated as it proceeds through the job life cycle. The file is updated again when the job reached either job-failed or job-complete status.

As part of the final job state transition, the _context.json file is is indexed into Elasticsearch. The file includes: job priority, job specs, parameters, etc.

_met.json 

This file is specific to HySDS, and includes three basic elements of a dataset, such as: start time, end time, and location.

_dataset.json

This file includes: label, start time, and version.