Context:
Via Alex Dunn regarding a HySDS pge he maintains:
Due to how the PGE runs, there will be an overlap in the products it creates with products that are already in the system. In this case, the PGE will error out with a no-clobber error instead of completing.
Is there a way to enable clobbering for a particular job type or what have you?
I can’t see anything in the job-spec specs.
Answer:
This is not an answer to the question, but a bit of background as to where the no-clobber error message comes from.
Osaka is the HySDS module that handles dataset localization and publishing; it abstracts the various backends that HySDS can write to like s3, azure, etc. By default, it is designed not to overwrite a dataset that already exists at the location. This behavior is controlled by the no-clobber parameter, which is default set to True.
The complication regarding the question above is that dataset publication is usually not handled directly by the PGE, but by the verdi worker as a post-processing step after the PGE executes. Is there a way to communicate to the verdi worker that for this PGE, we do want the no-clobber parameter to be set to False? Or is the only way to override this behavior to avoid using the verdi automatic publication step and have the PGE handle the dataset publication?