README.md

Ingest Envoy: The Workspaces NGAS & Metadata Ingestion System

Ingest Envoy is responsible for setup and launch of all types of file ingestion for the Workspaces System. Currently, this includes standard calibration and standard image ingestion.

usage: ingest_envoy [-h] [--calibration CALIBRATION] [--image IMAGE]

Workspaces Ingestion System

optional arguments:
  -h, --help            show this help message and exit
  --calibration CALIBRATION
                        run ingestion for a calibration product
  --image IMAGE         run ingestion for an image product

Ingest Envoy makes use of the existing ingest functionality of the AAT-PPI which simply takes an ingestion manifest as input. While this is consistent regaurdless of ingestion type, the manifest itself, as well as the ingestion staging requirements differ between the types of files to be ingested. For this reason, Ingest Envoy's functionality can be broken into two underlying parts: Setup and Launch.

Setup

Setup can be further divided into two essential components: Product Staging and Manifest Generation.

Product Staging

Product Staging is the step that collects all ingestable products and places them in the workspaces staging area located at: /lustre/aoc/cluster/pipeline/<capo-profile>/workspaces/staging

This collection is most often performed via a shell script such as the calibration-table-collector.sh for calibration ingestion or the image-product-collector.sh for image ingestion.

In the case of calibration ingestion, the collection script creates a tar file containing all the calibration tables and then creates a new weblog tar file to ensure that only the most recent version is ingested with the tables. Both tar files are then placed in the staging area.