Skip to content
Snippets Groups Projects

restore capability and workflow

Merged Charlotte Hausman requested to merge Restore_Capability into main
All threads resolved!
1 file
+ 66
0
Compare changes
  • Side-by-side
  • Inline
# CASA Envoy: The Workspaces CASA Processing System
CASA Envoy is responsible for environment setup and launch of all types of CASA processing for the Workspaces System.
Currently, this includes Standard Calibration, CMS Restores, Standard CMS Imaging, and CMS Restore-and-Image.
```
usage: casa_envoy [-h] [-c STANDARD_CAL STANDARD_CAL] [-i STANDARD_IMG STANDARD_IMG] [--restore] [--integrated]
Workspaces CASA processing launcher
optional arguments:
-h, --help show this help message and exit
-c, --standard-cal run the standard calibration CASA pipeline
-i, --standard-img run the standard imaging CASA pipeline
--restore run the restore measurement set CASA pipeline, use in conjunction with '-c'
--integrated run an integrated calibration-imaging pipeline, use in conjunction with '-i'
```
While initiating CASA is relatively straight forward and essentially identical between use cases, there are variations in
the required PPR structure and the initial location of input data. For this reason CASA Envoy's functionality
can be broken into two underlying parts: Setup and Launch.
## Setup
Setup can be further divided into three essential parts: Auditing, Environment, and Data
### Auditing
There are two types of auditing that occur in CASA Envoy. The first, Directory Auditing, ensures that the required
directory structure of *rawdata*, *working*, and *products* is contained in the processing directory.
The second type is File Auditing. CASA Envoy must be provided to different files at inital call, *metadata.json* and *PPR.xml*. These files must be
audited to ensure that all required fields are present. *PPR.xml* must be submitted directly to CASA so it is important
to make sure that it is sstructured correctly before submission. While *metadata.json* is not submitted to CASA, it does
contain all the information for results delivery post CASA and needs to be audited as well.
A further function of the File Auditing is correcting *PPR.xml* for processing with HTCondor. CASA requires the name of
the processing directory it is running in - unfortunately, with HTCondor it isn't possible to know that name prior to
submitting a condor job. Therefore, *PPR.xml* is corrected after submission to condor with the correct directory
information. This corrected *PPR.xml* is placed into the *working* directory where it is then used by CASA, and the
unaltered original remains in the parent processing directory.
### Environment
CASA has several environment variable that are required to be set by whatever system it is running on.
These include:
| ENV Variables | Description |
|--- |--- |
| SCIPIPE_ROOT | location of processing directory |
| CASA_HOME | location of CASA version to use for processing |
| PPR_FILENAME | name of PPR file (Should always be PPR.xml, but sometimes might not) |
| LANG | Turns out condor doesn't set this right for CASA happiness so we're fixing that here |
### Data
Usually the starting location for data is *rawdata*. This is the location where data is always placed by the
Workspaces Product Fetcher. For Standard Calibrations and Standard CMS Imaging this is correct. However, for
calibration restores, the calibration tar file needs to be extracted into the *products* directory. This step ensures
that all input data is in the correct location pre-processing for CASA to find when required.
## Launch
Since the only difference between calibration and imaging processing is the contents of *PPR.xml*, CASA Envoy contains
a single CASA Launcher class which is utilized by the two type launcher classes: Calibration Launcher and Imaging
Launcher. Each type launcher handles both standard and restore or integration types of processing. The two type classes ensure that setup is correct for each product type, as described above, and then calls
the CASA Launcher.
<br/>
<br/>
Post CASA processing, the casa log is checked for error flags and CASA envoy exits.
Loading