Skip to content
Snippets Groups Projects

Matrix architecture design documentation

Merged Charlotte Hausman requested to merge matrix-architechture into 2.8.4-DEVELOPMENT
2 unresolved threads
1 file
+ 18
17
Compare changes
  • Side-by-side
  • Inline
@@ -43,8 +43,8 @@ NRAO Archive
The archive allows a number of types of user driven processing, such as restoring calibrated data and reprocessing
images with certain inputs. Users are allowed to select a CASA version from the archive dropdowns in such cases. The
Matrix should provide the OPV as the recommended version to the user unless the version is no longer valid, in which
case it should recommend the default for the desired type of processing.
Matrix should provide the original processing version (OPV) as the recommended version to the user unless the version is
no longer valid, in which case it should recommend the default for the desired type of processing.
Workspaces Capability Service
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -57,7 +57,7 @@ ways:
1. Validating requested CASA Versions as with the Archive above
2. Dynamically determining and rendering standard CASA procedure recipies for PPR.xml's
2. Dynamically determining and rendering standard CASA procedure recipes for PPR.xml's
For requests originating from the Archive, only number 2 will be relevant as number 1 was handled prior to request
submission.
@@ -68,12 +68,12 @@ What are the Matrix's responsibilities?
As the CASA Version Matrix is to be considered the one source of truth for what versions of CASA are valid for processing,
the Matrix has a number of responsibilities. These include, but are not necessarily limited to, tracking of what
versions+pipelines are capable of running on the current NRAO clusters, tracking of what versions+pipelines support a
given processing capability, tracking and access of processing recipies for PPR generation, and maintenance of SSA's
given processing capability, tracking and access of processing recipes for PPR generation, and maintenance of SSA's
CASA processing links.
Maintenance of SSA's CASA Links
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
In order for us to proved access to, and processing of, ALMA data in a timely fashion, we run all ALMA processing workflows
In order for us to provide access to, and processing of, ALMA data in a timely fashion, we run all ALMA processing workflows
on the NRAO Charlottesville cluster, where the North American ALMA Science Center (NAASC) is located and our copy of ALMA's
rawdata is easily accessible. Otherwise, we'd have to steam the data to DSOC, and that slows everything down.
@@ -97,14 +97,14 @@ What makes a CASA version valid?
________________________________
In order for a CASA version to be valid for processing the following conditions must be met:
1. it *must* be capable of running on the current cluster's operating system.
1. It *must* be capable of running on the current cluster's operating system.
- All processing currently occurs within one of NRAO's processing clusters. In the future we plan to enable fully remote processing, but at this time we are limited by the conditions of our clusters' resources. The current clusters are running RHEL8; we have had issues running 6.1.x versions on this OS. These versions are considered invalid for processing.
2. It *must* be installed and linked to the SSA home area.
- Even if the requested version can run on the current cluster, if it's not installed in the expected area and linked to SSA home, it doesn't exist as far as processing is concerned.
3. It *must* be able to run the requested type of processing.
- As time passes, CASA surfaces new processing abilities. Even if a CASA version meets the other validity criteria, if it predates the start of the requested type, it's useless, as the needed recipie isn't available. For example: EVLA Standard Imaging only began support around CASA 5.4.x. Any version predating that is invalid for Standard Imaging.
- As time passes, CASA surfaces new processing abilities. Even if a CASA version meets the other validity criteria, if it predates the start of the requested type, it's useless, as the needed recipe isn't available. For example: EVLA Standard Imaging only began support around CASA 5.4.x. Any version predating that is invalid for Standard Imaging.
Tracking of Original Processing Versions (OPV)
______________________________________________
@@ -120,7 +120,7 @@ existing calibration products to fill in their missing version and pipeline info
What if a OPV version isn't valid?
__________________________________
The first this the matrix will do is determine if the requested product has OPV information available in the database.
The first thing the matrix will do is determine if the requested product has OPV information available in the database.
If the product has version information, the Matrix will then verify if the OPV still meets the validity criteria
described above. Does it run on the current cluster? Is it installed in SSA home? Is it valid for the requested type of reprossesing?
@@ -134,7 +134,7 @@ a list of usable CASA versions and known issues, found here: `Pipeline Version H
In the case where the OPV is *not* valid, **the Matrix shall fallback to the CAPO defined default for the requested
processing type**.
If the CAPO define default is *also* not valid, **the Matrix shall fallback to the newest available version**.
If the CAPO defined default is *also* not valid, **the Matrix shall fallback to the newest available version**.
Concisely, the version precedence chain is as follows:
@@ -148,23 +148,24 @@ System specified Defaults
_________________________
There are a number of system capability defaults specified by CAPO properties for ease and consistency with current systems.
It is possible it set processing specific defaults for Basic Restores, CMS Restores, AUDI, VUDI, and Standard
It is possible to set processing specific defaults for Basic Restores, CMS Restores, AUDI, VUDI, and Standard
Calibration/Imaging (they use the same default).
All CASA version defaults use the *edu.nrao.archive.workflow.config.CasaVersions* prefix and start with *homeFor*.
All CASA version defaults use the *edu.nrao.archive.workflow.config.CasaVersions* prefix and start with *homeFor*. Each
capability defines which CAPO property to use as its default at the database level.
Dynamic PPR Generation
^^^^^^^^^^^^^^^^^^^^^^
As the Matrix is responsible for handling all things related to CASA versions used by our processing infrastructure, it
shall also handle the determination, location, and provision of CASA processing recipies.
shall also handle the determination, location, and provision of CASA processing recipes.
Each CASA pipeline ships with a set of standard recipie templates for various types of processing. These can change
between releases, therefore in order to provide support for multiple versions we must be able to pull the needed recipie
Each CASA pipeline ships with a set of standard recipe templates for various types of processing. These can change
between releases, therefore in order to provide support for multiple versions we must be able to pull the needed recipe
from the requested version as needed.
After verifying the requested version's validity, the Matrix shall determine if a recipie exists for the requested type
of processing. If a recipie should exist, the Matrix shall then attempt to locate the needed file in the CASA version
After verifying the requested version's validity, the Matrix shall determine if a recipe exists for the requested type
of processing. If a recipe should exist, the Matrix shall then attempt to locate the needed file in the CASA version
on disk in the cluster and return it to the calling service (most likely the WS Capabiltiy Service) as text formatted for
rendering in a mustache template. If the expected recipie file cannot be found, the Matrix shall return an empty object
rendering in a mustache template. If the expected recipe file cannot be found, the Matrix shall return an empty object
and inform the calling system to use the hardcoded default template (this should be current with the most recent CASA version).
Loading