Conveyor: The Workspaces Quality Assurance Delivery/Retrieval System

Conveyor is responsible for delivery to, and retrieval from, the Workspaces Quality Assurance area on lustre. It is also responsible for maintaining the linkages between the QA directories and the Workspaces parent processing directory during the QA process.

Capabilities which require QA are handled differently upon workflow completion. Where other requests are delivered directly to the user for download, requests requiring QA are delivered to the QA area on lustre where they wait for a Data Analysts to inspect them. Once the request has been either QA Passed or QA Failed, the data is retrieved from the QA area back to the Workspaces parent processing directory for either post fail operations or ingestion into the archive.

Capabilities which require QA include, but are not limited to, Standard Calibration and Standard Imaging.

Usage

usage: conveyor [-h] [--deliver DELIVER] [--retrieve RETRIEVE] [--deliver-img DELIVER_IMG] [--retrieve-img RETRIEVE_IMG] [--qa-fail QA_FAIL]

Workspaces Standard Capability Postprocessing

optional arguments:
  -h, --help            show this help message and exit
  --deliver DELIVER     deliver standard calibration directories to analyst QA area
  --retrieve RETRIEVE   return standard calibration directories to original parent directory
  --deliver-img DELIVER_IMG
                        deliver standard image results to analyst image QA area
  --retrieve-img RETRIEVE_IMG
                        retrieve standard image results to original parent directory
  --qa-fail QA_FAIL     clean up after a QA Fail

Conveyor only requires a single input regardless of the type of conveyance chosen - The workflow's metadata.json file. This file holds all information needed to keep track of the original directories of the request.

QA Delivery

Once a workflow is complete, if the capability has a QA step, conveyor caches and unzips the weblog tar file, and then moves the rawdata, products, and working directories from the parent processing directory in the Workspaces spool area into the appropriate Workspaces QA area - either workspaces/qa2 or workspaces/image-qa.

Conveyor then creates symlinks for each moved directory between their parent processing directory and the new qa directory. This allows all files produced by a workflow to remain continuously accessible no matter their location. This was particularly important for viewing the CASA weblog from the request status page.

QA Retrieval

Once the DAs and/or AOD are finished with the QA process, the files and directories need to be moved back to their original parent processing directory.

Conveyor first checks for and break any symlinks, and then moves the requested directories back to their original location in spool. This is the first stage of any ingestion workflow and will also occur for QA Fail, however that is not yet implemented.

QA Failure

When a calibration has been failed (qa-fail), the files and directories need to be moved back to their original parent processing directory. Conveyer uses RetrievalConveyor system for this.