Matrix architecture design documentation

Daniel wrote up a good usage document in the README, this is to document the design decisions that the code should support. Please point out anything I missed

shared/workspaces/workspaces/system/ARCHITECTURE.rst

+CASA Version Matrix Architecture
+================================
+The CASA Version Matrix is intended to provide SSA's data processing infrastructure with dynamic support of CASA versions: past,
+present, and future. As CASA has evolved and grown, the ways in which it runs processing have changed. In order to
+provide support for user driven processing, in which a user may choose to use any CASA version available, it has become
+necessary to define a system that can validate, verify, and adapt to changes between various CASA versions and our own
+infrastructure.
+
+A user is allowed to select their desired CASA version for most types of processing surfaced through the Archive. At
+this time, we support the following types of user driven processing:
+
+Archive
+
+* Basic Restores
+* CMS Restores
+* ALMA User Driven Imaging (AUDI) **
+
+**  AUDI currently only allows use of the specified Default version
+
+What is CASA?
+------------
+
+Common Astronomy Software Applications (CASA) is a radio astronomy data processing package built and released by NRAO
+and our associated partner institutions.
+It is used for all standard, and user driven, data processing for observation data coming out of the EVLA and ALMA
+telescopes. CASA is the primary processing package supported by the NRAO
+Archive at this time and all our data processing workflows rely on it.
+
+It is important to note that there are actually two different types of CASA releases we need to pay attention to. The
+CASA release by itself, and the CASA release with a packaged pipeline. We are only concerned with the pipeline releases,
+as they are what we use from within workflows. Please be aware that when we refer to a 'CASA version' within the Matrix,
+we are referring to the **CASA version+pipeline** rather than the standalone CASA release.
+
+More information on CASA can be found at the official website here: `<https://casa.nrao.edu>`_
+
+Interacting Systems
+-------------------
+
+There are two major systems that will interact with the CASA Version Matrix at this time.
+
+NRAO Archive
+^^^^^^^^^^^^
+
+The archive allows a number of types of user driven processing, such as restoring calibrated data and reprocessing
+images with certain inputs. Users are allowed to select a CASA version from the archive dropdowns in such cases. The
+Matrix should provide the original processing version (OPV) as the recommended version to the user unless the version is
+no longer valid, in which case it should recommend the default for the desired type of processing.
+
+Workspaces Capability Service
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+While the CASA Version Matrix is served from within the WS Capability Service docker container, it is treated as a
+separate system, as with the Archive Service which handles automatic calibration submission based on ingestion
+completion.
+
+For requests originating from within the Workspaces system, the Capability Service will interact with the Matrix in two
+ways:
+
+1. Validating requested CASA Versions as with the Archive above
+
+2. Dynamically determining and rendering standard CASA procedure recipes for PPR.xml's
+
+For requests originating from the Archive, only number 2 will be relevant as number 1 was handled prior to request
+submission.
+
+
+What are the Matrix's responsibilities?
+---------------------------------------
+As the CASA Version Matrix is to be considered the one source of truth for what versions of CASA are valid for processing,
+the Matrix has a number of responsibilities. These include, but are not necessarily limited to, tracking of what
+versions+pipelines are capable of running on the current NRAO clusters, tracking of what versions+pipelines support a
+given processing capability, tracking and access of processing recipes for PPR generation, and maintenance of SSA's
+CASA processing links.
+
+Maintenance of SSA's CASA Links
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+In order for us to provide access to, and processing of, ALMA data in a timely fashion, we run all ALMA processing workflows
+on the NRAO Charlottesville cluster, where the North American ALMA Science Center (NAASC) is located and our copy of ALMA's
+rawdata is easily accessible. Otherwise, we'd have to steam the data to DSOC, and that slows everything down.
+
+The standard NRAO CASA installs at each site (DSOC vs NAASC) are not kept in locations with the same
+access paths. As SSA does not, and has no desire to, control the install locations of CASA, we maintain our own area in
+each site's processing center with symlinks to the CASA installations which are relevant to our systems.
+This area is */home/ssa/casa/* and we sync the links form DSOC to NAASC regularly to provide a standard access point
+across NRAO sites.
+
+Under the current Archive system, these links are created and maintained by a cronjob running a bash script which was
+last updated in 2020. The Matrix is designed to subsume this script, replacing it with a service endpoint, and possibly
+replace the cron job in the future.
+
+Determination of Valid CASA Versions
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+In order to provide our users with the most accurate data products possible, we need to make every effort to
+ensure their processing requests succeed. As data processing results depend on the outcome of CASA, we must verify the
+CASA version used to be as correct as possible for their requests.
+
+What makes a CASA version valid?
+________________________________
+In order for a CASA version to be valid for processing the following conditions must be met:
+
+1. It *must* be capable of running on the current cluster's operating system.
+    - All processing currently occurs within one of NRAO's processing clusters. In the future we plan to enable fully remote processing, but at this time we are limited by the conditions of our clusters' resources. The current clusters are running RHEL8; we have had issues running 6.1.x versions on this OS. These versions are considered invalid for processing.
+
+2. It *must* be installed and linked to the SSA home area.
+    - Even if the requested version can run on the current cluster, if it's not installed in the expected area and linked to SSA home, it doesn't exist as far as processing is concerned.
+
+3. It *must* be able to run the requested type of processing.
+    - As time passes, CASA surfaces new processing abilities. Even if a CASA version meets the other validity criteria, if it predates the start of the requested type, it's useless, as the needed recipe isn't available. For example: EVLA Standard Imaging only began support around CASA 5.4.x. Any version predating that is invalid for Standard Imaging.
+
+Tracking of Original Processing Versions (OPV)
+______________________________________________
+
+All types of processing currently serviced from the Archive represent a reprocessing of existing, ingested, data products
+(i.e. calibrations or images). The primary case we're concerned with at this time is Restores.
+
+Ideally, we should be reproducing these products using the *same* CASA version the product was originally produced by.
+Until now we have not been retaining the CASA version used upon ingestion, despite already having an existing database field for calibrations.
+With the implementation of the Matrix, we will also begin ingestion of the CASA version and pipeline used to produce
+the product in question, as this is essential information for Matrix functionality. We will also be back-populating
+existing calibration products to fill in their missing version and pipeline information.
+
+What if a OPV version isn't valid?
+__________________________________
+The first thing the matrix will do is determine if the requested product has OPV information available in the database.
+If the product has version information, the Matrix will then verify if the OPV still meets the validity criteria
+described above. Does it run on the current cluster? Is it installed in SSA home? Is it valid for the requested type of reprossesing?
+
+That last question might seem a bit strange. Clearly the product was produced using this version in the first place, so
+shouldn't that mean the version *must* be valid for reprocessing of this product? Simply, no. Just because the OPV
+was used to produce the product, does not guarantee it can successfully reproduce it.
+There are known CASA versions which might have been used to produce calibrations which passed QA and were ingested,
+which were only later determined to have processing errors, rendering the version unsuitable. The Data Analysts maintain
+a list of usable CASA versions and known issues, found here: `Pipeline Version History <https://science.nrao.edu/facilities/vla/data-processing/pipeline/pipeline-version-history>`_
+
+In the case where the OPV is *not* valid, **the Matrix shall fallback to the CAPO defined default for the requested
+processing type**.
+
+If the CAPO defined default is *also* not valid, **the Matrix shall fallback to the newest available version**.
+
+Concisely, the version precedence chain is as follows:
+
+1. The Original Processing Version shall be preferred above all else
+
+2. If the OPV is not valid, use the CAPO defined default
+
+3. If the default is not valid, use the most current installed version
+
+System specified Defaults
+_________________________
+
+There are a number of system capability defaults specified by CAPO properties for ease and consistency with current systems.
+It is possible to set processing specific defaults for Basic Restores, CMS Restores, AUDI, VUDI, and Standard
+Calibration/Imaging (they use the same default).
+
+All CASA version defaults use the *edu.nrao.archive.workflow.config.CasaVersions* prefix and start with *homeFor*. Each
+capability defines which CAPO property to use as its default at the database level.
+
+
+Dynamic PPR Generation
+^^^^^^^^^^^^^^^^^^^^^^
+As the Matrix is responsible for handling all things related to CASA versions used by our processing infrastructure, it
+shall also handle the determination, location, and provision of CASA processing recipes.
+
+Each CASA pipeline ships with a set of standard recipe templates for various types of processing. These can change
+between releases, therefore in order to provide support for multiple versions we must be able to pull the needed recipe
+from the requested version as needed.
+
+After verifying the requested version's validity, the Matrix shall determine if a recipe exists for the requested type
+of processing. If a recipe should exist, the Matrix shall then attempt to locate the needed file in the CASA version
+on disk in the cluster and return it to the calling service (most likely the WS Capabiltiy Service) as text formatted for
+rendering in a mustache template. If the expected recipe file cannot be found, the Matrix shall return an empty object
+and inform the calling system to use the hardcoded default template (this should be current with the most recent CASA version).