Skip to content
Snippets Groups Projects

Implemented manifest generation for EB full curation in ingest_envoy

Merged Sam Kagan requested to merge teach-ingest_envoy-full-curation into 2.8.2.1-DEVELOMENT

Along the way:

  • Added tests for partial-curation manifest generation for ingest_envoy
  • Separated IngestionManifestBuilder's directory-to-search (staging_source_dir for ingestion, curation_source for curation) from the directory it wants to put its manifest in since partial curation doesn't always have the former.
Edited by Sam Kagan

Merge request reports

Merge request pipeline #12753 passed

Merge request pipeline passed for c11a0c49

Approved by

Merged by Sam KaganSam Kagan 1 year ago (Nov 9, 2023 5:10pm UTC)

Pipeline #12755 passed

Pipeline passed for 6ffc56b5 on 2.8.2.1-DEVELOMENT

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
302 302 self.logger = logging.getLogger("ingest_envoy")
303 303 self.curator_type = arg_type
304 304 self.sci_product_type = parameters["product_type"]
305 self.curation_source = parameters["curation_source"]
306 # Curation doesn't need a curation_source, but it does need a place to stick the manifest
307 self.manifest_dir = Path(self.curation_source) if self.curation_source is not None else Path.cwd()
  • 336 356
    337 357 return manifest, manifest_file
    338 358
    359 def _build_full_curation_manifest(self) -> (IngestionManifest, Path):
  • 278 283 raise NotImplementedError(f"Don't know yet how to build a {self.sp_type.value} manifest")
    279 284
    280 285 if self.curation_type is not None:
    281 if self.curation_type not in [CuratorType.PARTIAL]:
    286 if self.curation_type not in [CuratorType.PARTIAL, CuratorType.FULL]:
    282 287 raise NotImplementedError(
    283 288 f"Don't know how to build a {self.curation_type.value} curation {self.sp_type} manifest"
    284 289 )
    285 290 self.curation_source = Path(curate[1]) if curate and curate[1] else None
    286 291 self.target_list = curate[2]
    292 if self.curation_type == CuratorType.FULL and self.target_list != None:
  • Daniel Nemergut
  • Sam Kagan added 2 commits

    added 2 commits

    • 58ffddf3 - Removed dead code
    • 6c3fb9eb - Renamed manifest_dir and source_dir, made full-cur targets check less strict

    Compare with previous version

  • Sam Kagan added 1 commit

    added 1 commit

    • af979aa1 - Failed and passed tests for requiring man_dest_dir for curation and for writing manifest to it

    Compare with previous version

  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Please register or sign in to reply
    Loading