Hello Dataset- Installation- Github
torresal edited this page on May 24, 2018 · 8 revisions
Confidence Level TBD This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it. |
---|
Integrating a "Hello World" Dataset Type
For definitions of terminology used, please refer to our terminology reference.
In this tutorial, we continue where we left off in integrating our first job type, Hello World. We will update our job type to generate a HySDS dataset that will be automatically ingested and catalogued into the system.
On the
mozart
instance, navigate to thehello_world
repositorycd ~/mozart/ops/hello_world
We need to author a dataset ID scheme according to the HySDS dataset specification. In this tutorial, we will use the following dataset ID scheme
hello_world-product-<year><month><day>T<hour><minute><second>Z-<hash>
where
<year>
will be replaced by a fictional observation year and likewise for<month>
,<day>
,<hour>
,<minute>
,<second>
and<hash>
. An example instance of this dataset ID would behello_world-product-20170901T000102.021Z-a3d0x
Edit the script that runs our job type
and add code to create an instance of our dataset. The script should look like this
Access the documentation on the specifications for the HySDS dataset JSON file and metadata JSON file for more detail. Finally, commit your change and push the update
If your jenkins job for the master branch of your
hello_world
repo was configured correctly, thegit push
should have kicked off a rebuild of the container. If not, navigate to yourci
instance (e.g. http://:8080) and manually schedule a buildTest the script by running it
You should see the dataset directory you just created sitting there in your directory. Ensure that the
fake_data.dat
and the HySDS dataset and metadata JSON files were generatedNow let's try to ingest this new dataset into our catalog
You should've gotten an error that the data was not recognized
Let's configure our new dataset so that it can be recognized by our HySDS cluster. Since we want to ensure all nodes in our cluster is able to recognize this new dataset, we need to make updates to the
datasets.json
file in~/.sds/files
. Still onmozart
edit the datasets.json file
and add the following dataset configuration below the
dumby-product
configurationand save.
Run the following fabric command to push the update you made to the
datasets.json
template onmozart
You should have output similar to the following
Now let's try to ingest this new dataset into our catalog
Verify that your new dataset was published to your HySDS cluster. Open up a browser and point it to your
grq
instance'stosca
(Dataset FacetSearch) interface (e.g. https:///search). On the left panel you should see your new dataset type listed under the "dataset" facet showing 1 resultClick on "hello_world-product" under the "dataset" facet and your result set should be constrained now to just the dataset you just ingested
Now that we've verified that the new dataset is recognized by the system, let's push the updates to the rest of the system. On
mozart
That's it!
Congratulations, you've integrated a new dataset type into your HySDS cluster.
To configure Autoscaling groups for your HySDS cluster, continue on to Create-AWS-Autoscaling-Group-for-Verdi.
To configure a staging area for your HySDS cluster, continue on to Create-AWS-Resources-for-Staging-Area.
Related Articles: |
---|
Have Questions? Ask a HySDS Developer: |
Anyone can join our public Slack channel to learn more about HySDS. JPL employees can join #HySDS-Community
|
JPLers can also ask HySDS questions at Stack Overflow Enterprise
|
Page Information: |
---|
Was this page useful? |
Contribution History:
|
Subject Matter Expert: @Alexander Torres |
Find an Error? Is this document outdated or inaccurate? Please contact the assigned Page Maintainer: @Lan Dang |