Implemented manifest generation for EB full curation in ingest_envoy
Along the way:
- Added tests for partial-curation manifest generation for ingest_envoy
- Separated IngestionManifestBuilder's directory-to-search (staging_source_dir for ingestion, curation_source for curation) from the directory it wants to put its manifest in since partial curation doesn't always have the former.
Edited by Sam Kagan
Merge request reports
Activity
added 40 commits
-
47ef4752...f6a0a461 - 38 commits from branch
2.8.2.1-DEVELOMENT
- e5d3479d - Added failing tests for creating full-curation manifest
- 603850b6 - Passed tests for ingest_envoy full-curation EB manifests
-
47ef4752...f6a0a461 - 38 commits from branch
added 1 commit
- 1a179259 - Greatly condensed image manifest tests in ingest_envoy
added 1 commit
- 1c1f9139 - Created failing tests for full-curation image man. creation
302 302 self.logger = logging.getLogger("ingest_envoy") 303 303 self.curator_type = arg_type 304 304 self.sci_product_type = parameters["product_type"] 305 self.curation_source = parameters["curation_source"] 306 # Curation doesn't need a curation_source, but it does need a place to stick the manifest 307 self.manifest_dir = Path(self.curation_source) if self.curation_source is not None else Path.cwd() You didn't actually change this. It's still wanting to put the manifest in the wrong place for curation. Since curation is not interacting with NGAS, which would necessitate using the staging area, all files should be written to the cwd, which should be the processing directory of the workflow in spool.
changed this line in version 12 of the diff
336 356 337 357 return manifest, manifest_file 338 358 359 def _build_full_curation_manifest(self) -> (IngestionManifest, Path): changed this line in version 10 of the diff
278 283 raise NotImplementedError(f"Don't know yet how to build a {self.sp_type.value} manifest") 279 284 280 285 if self.curation_type is not None: 281 if self.curation_type not in [CuratorType.PARTIAL]: 286 if self.curation_type not in [CuratorType.PARTIAL, CuratorType.FULL]: 282 287 raise NotImplementedError( 283 288 f"Don't know how to build a {self.curation_type.value} curation {self.sp_type} manifest" 284 289 ) 285 290 self.curation_source = Path(curate[1]) if curate and curate[1] else None 286 291 self.target_list = curate[2] 292 if self.curation_type == CuratorType.FULL and self.target_list != None: changed this line in version 10 of the diff
- Resolved by Sam Kagan
added 1 commit
- af979aa1 - Failed and passed tests for requiring man_dest_dir for curation and for writing manifest to it
Please register or sign in to reply