CASA Envoy: The Workspaces CASA Processing System
CASA Envoy is responsible for environment setup and launch of all types of CASA processing for the Workspaces System. Currently, this includes Standard Calibration, CMS Restores, Standard CMS Imaging, and CMS Restore-and-Image.
usage: casa_envoy [-h] [-c STANDARD_CAL STANDARD_CAL] [-i STANDARD_IMG STANDARD_IMG] [--restore] [--integrated]
Workspaces CASA processing launcher
optional arguments:
-h, --help show this help message and exit
-c, --standard-cal run the standard calibration CASA pipeline
-i, --standard-img run the standard imaging CASA pipeline
--restore run the restore measurement set CASA pipeline, use in conjunction with '-c'
--integrated run an integrated calibration-imaging pipeline, use in conjunction with '-i'
While initiating CASA is relatively straight forward and essentially identical between use cases, there are variations in the required PPR structure and the initial location of input data. For this reason CASA Envoy's functionality can be broken into two underlying parts: Setup and Launch.
Setup
Setup can be further divided into three essential parts: Auditing, Environment, and Data
Auditing
There are two types of auditing that occur in CASA Envoy. The first, Directory Auditing, ensures that the required directory structure of rawdata, working, and products is contained in the processing directory.
The second type is File Auditing. CASA Envoy must be provided two different files at inital call, metadata.json and PPR.xml. These files must be audited to ensure that all required fields are present. PPR.xml must be submitted directly to CASA, so it is important to make sure that it is structured correctly before submission. While metadata.json is not submitted to CASA, it does contain all the information for results delivery post-CASA and needs to be audited as well.
A further function of File Auditing is correcting PPR.xml for processing with HTCondor. CASA requires the name of the processing directory it is running in - unfortunately, with HTCondor it isn't possible to know that name prior to submitting a condor job. Therefore, PPR.xml is corrected after submission to condor with the correct directory information. This corrected PPR.xml is placed into the working directory where it is then used by CASA, and the unaltered original remains in the parent processing directory.
Environment
CASA has several environment variables that are required to be set by whatever system it is running on. These include:
| ENV Variables | Description |
|---|---|
| SCIPIPE_ROOT | location of processing directory |
| CASA_HOME | location of CASA version to use for processing |
| PPR_FILENAME | name of PPR file (Should always be PPR.xml, but sometimes might not) |
| LANG | Turns out condor doesn't set this right for CASA happiness so we're fixing that here |
Data
Usually the starting location for data is rawdata. This is the location where data is always placed by the Workspaces Product Fetcher. For Standard Calibrations and Standard CMS Imaging this is correct. However, for calibration restores, the calibration tar file needs to be extracted into the products directory. This step ensures that all input data is in the correct location pre-processing for CASA to find when required.
Launch
Since the only difference between calibration and imaging processing is the contents of PPR.xml, CASA Envoy contains a single CASA Launcher class which is utilized by the two typed launcher classes: Calibration Launcher and Imaging Launcher. Each type launcher handles both standard and restore or integration types of processing. The two type classes ensure that setup is correct for each product type, as described above, and then calls the CASA Launcher.
Post CASA processing, the casa log is checked for error flags and CASA envoy exits.