Skip to content
GitLab
Explore
Sign in
Register
Primary navigation
Search or go to…
Project
workspaces
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Wiki
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Build
Pipelines
Jobs
Pipeline schedules
Artifacts
Deploy
Releases
Package Registry
Model registry
Operate
Environments
Terraform modules
Monitor
Incidents
Analyze
Value stream analytics
Contributor analytics
CI/CD analytics
Repository analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Terms and privacy
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
ssa
workspaces
Merge requests
!313
WS-507: More foundation-laying for ingestion manifest builder
Code
Review changes
Check out branch
Download
Patches
Plain diff
Merged
WS-507: More foundation-laying for ingestion manifest builder
WS-507-build-ingestion-manifest
into
main
Overview
1
Commits
2
Pipelines
6
Changes
9
1 unresolved thread
Hide all comments
Merged
Janet Goldstein
requested to merge
WS-507-build-ingestion-manifest
into
main
3 years ago
Overview
1
Commits
2
Pipelines
6
Changes
9
1 unresolved thread
Hide all comments
Expand
Fleshed out classes
Added example files
Got started on first test
stole lots from the archive version
Edited
3 years ago
by
Janet Goldstein
0
0
Merge request reports
Compare
main
version 5
4c71fb78
3 years ago
version 4
9e369589
3 years ago
version 3
7206f614
3 years ago
version 2
0d65945a
3 years ago
version 1
c967cef9
3 years ago
main (base)
and
version 2
latest version
2ef91698
2 commits,
3 years ago
version 5
4c71fb78
3 commits,
3 years ago
version 4
9e369589
8 commits,
3 years ago
version 3
7206f614
7 commits,
3 years ago
version 2
0d65945a
6 commits,
3 years ago
version 1
c967cef9
4 commits,
3 years ago
9 files
+
389
−
1
Inline
Compare changes
Side-by-side
Inline
Show whitespace changes
Show one file at a time
Files
9
Search (e.g. *.vue) (Ctrl+P)
apps/cli/executables/pexable/ingest_envoy/ingest_envoy/ingestion_manifest.py
0 → 100644
+
83
−
0
Options
"""
The ingestion manifest
"""
from
pathlib
import
Path
from
.utility_classes
import
(
Telescope
,
ScienceProduct
,
)
class
Parameters
:
"""
a manifest
'
s various input parameters
"""
def
__init__
(
self
,
telescope
:
Telescope
,
ingestion_path
:
Path
,
additional_metadata
:
str
,
collection_metadata
:
str
,
reingest
:
bool
=
False
,
ngas_ingest
:
bool
=
False
,
):
self
.
telescope
=
telescope
self
.
ingestion_path
=
ingestion_path
self
.
additional_metadata
=
additional_metadata
self
.
collection_metadata
=
collection_metadata
self
.
reingest
=
reingest
self
.
ngas_ingest
=
ngas_ingest
class
InputGroup
:
"""
This represents the starting point for processing which generated a science product.
There is not always an input group for every output group (rawdata, for instance).
Initial assumption: Input groups consist only of science products.
"""
def
__init__
(
self
):
self
.
science_products
=
[]
class
IngestionManifest
:
"""
Represents JSON layout of ingestion information, encompassing several potential scenarios.
see ingest_envoy/test/examples, nicked from https://open-confluence.nrao.edu/x/roPCAQ
"""
def
__init__
(
self
,
parameters
:
Parameters
):
self
.
parameters
=
parameters
# to be an InputGroup
self
.
input_group
=
None
# to be an OutputGroup
self
.
output_group
=
None
# to be an AssociateGroup (not required?)
self
.
associate_group
=
None
self
.
science_products
=
[]
self
.
ancillary_products
=
[]
class
OutputGroup
:
"""
Represents result of data processing
"""
def
__init__
(
self
):
self
.
science_products
=
[]
self
.
ancillary_products
=
[]
class
AssociateGroup
:
"""
A representation of Science Products which are not part of the same Input or Output groups
but are still fundamentally linked. Created for RealFast project, to link the RealFast
specific execution block & image to the execution block during which a transient was
discovered.
Associate groups also, by definition, include any science product(s) within the output
group to be ingested. The new locators generated at ingestion time will be added to any
which compose an associate group in the manifest.
"""
def
__init__
(
self
):
self
.
science_products
=
[]
Loading