Merge branch 'fix-qa-fail-template-custom-text' into 'main'

Update the QA pass template to work with non-srdp as well (ignore the branch/migration misnomer)

See merge request !1461

shared/workspaces/alembic/versions/d20ceed949b3_make_sure_qa_fail_custom_text_is_in_.py

+"""make sure qa fail custom text is in srdp and non srdp sections
+
+Revision ID: d20ceed949b3
+Revises: 569416c40ca8
+Create Date: 2023-09-29 12:27:27.449396
+
+"""
+from pathlib import Path
+
+import sqlalchemy as sa
+from alembic import op
+
+# revision identifiers, used by Alembic.
+revision = "d20ceed949b3"
+down_revision = "569416c40ca8"
+branch_labels = None
+depends_on = None
+
+template_name = "std_calibration_complete"
+old_content = (Path.cwd() / "versions" / "templates" / "emails" / "std_calibration_complete_newer.txt").read_text()
+new_content = (
+    Path.cwd() / "versions" / "templates" / "emails" / "std_calibration_complete_non_srdp_custom_text.txt"
+).read_text()
+
+
+def upgrade():
+    op.execute(
+        f"""
+        UPDATE notification_templates
+        SET template=E'{new_content}'
+        WHERE name=E'{template_name}'
+        """
+    )
+
+
+def downgrade():
+    op.execute(
+        f"""
+        UPDATE notification_templates
+        SET template=E'{old_content}'
+        WHERE name=E'{template_name}'
+        """
+    )

shared/workspaces/alembic/versions/templates/emails/std_calibration_complete_non_srdp_custom_text.txt

+{{#version}}{{#parameters}}{{#metadata}}{{#is_srdp}}Subject: Calibration complete for {{sdm_id}}
+
+Hello,
+
+One of your Scheduling Blocks,
+
+{{sdm_id}}, observed on {{obs_end_time}}, {{bands}}, <filesize>
+
+has been processed through the VLA CASA Calibration Pipeline using CASA 6.4.1, which is designed to handle Stokes I continuum data, and is now available should you wish to access the calibrated data.
+
+These results have been checked by NRAO staff and notes about the quality assurance are below.
+
+{{#custom_text}}{{custom_text}}{{/custom_text}}{{^custom_text}}The results of this data processing request passed our quality assurance, with the following notes:{{/custom_text}}
+
+The following notes are also in the weblog:
+
+{{#workflow_metadata}}{{#qa}}{{qa}}{{/qa}}{{^qa}}No QA notes available.{{/qa}}{{/workflow_metadata}}{{^workflow_metadata}}No QA notes available.{{/workflow_metadata}}
+
+Accessing Pipeline Products:
+
+There are two main pipeline products you may be interested in: the calibrated Measurement Set (MS) or the smaller pipeline product tar file that includes the weblog, final calibration and flag tables, a restore script, and other small files related to the calibration.
+
+For the entire Calibrated MS and pipeline tar file: you will need to request a calibrated MS (CMS) from our new archive access tool at https://data.nrao.edu . To access the pipeline products, please follow these steps:
+
+Click the "Log in" link at the top right of this page and select the NRAO login, which should take you to a page where you will enter your my.nrao.edu login information.
+
+Once logged in, you should be able to access your projects, both public and proprietary by selecting <yourusername>\'s data: navigate to the desired project and click the "+" symbol at the left to expand the list of available SBs.
+
+Find the specific observation you want and if you would like to recreate the pipeline calibration, click the "Add to clipboard" button to the left of that observation.
+
+Click the "Download" button at the top of the project list.
+
+In the pop-up window that opens, leave the default options the same: this should have the "Choose download data format:" option to "Calibrated Measurement Set", and the "Restore previous CMS" option should be filled in with a tar file named something like project-code_YYYY_MM_DD_THH_MM_SS.SSS.tar. Note that these two options are unavailable when more than one archive file is added to the clipboard. If the calibrated measurement set is needed for multiple scheduling blocks they must be downloaded one at a time. Since this calibration was done with CASA 6.2.1-7 leave the "CASA Version:" drop-down menu at this version. Now click the "Submit Request" button.
+
+Once ready, you should receive an email with download instructions. This restoration of the calibrated MS may take several hours or longer, depending on the specifics of your observation.
+
+=====
+
+For just the pipeline tar file (includes the weblog, final flag versions, final calibration tables, restore script, and other related small files)
+
+Click the "Log in" link at the top right of this page and select the NRAO login, which should take you to a page where you will enter your my.nrao.edu login information.
+
+Once logged in, you should be able to access your projects, both public and proprietary: navigate to the desired project and click the "+" symbol at the left to expand the list of available SBs.
+
+Find the specific observation you want and if you would like to recreate the pipeline calibration, click the icon in the "Cals" column near the right side of the returned list, which should bring up a pop-up window to download a tar file named something like project-code_YYYY_MM_DD_THH_MM_SS.SSS.tar, and click "Submit Request". Once ready, you should receive an email with download instructions.
+
+____________
+
+For more information about the pipeline, including instructions for rerunning the pipeline, applying pipeline calibration to raw data, or modification to suit your particular science goals, or access to the scripted pipeline please visit our pipeline web page: https://science.nrao.edu/facilities/vla/data-processing/pipeline
+
+For more information about the SRDP project, please see https://science.nrao.edu/srdp
+
+Please let us know if you have any questions or concerns through the NRAO Helpdesk (https://help.nrao.edu/) , using the VLA Pipeline department for questions about the pipeline processing, the VLA/VLBA Archive and Data Retrieval department for questions about data retrieval, and the VLA Data Products department for questions about quality assurance and the use of of Science-Ready (SRDP) products.{{/is_srdp}}{{^is_srdp}}Subject: Calibration complete for {{sdm_id}}
+
+Hello,
+
+One of your Scheduling Blocks has been processed through the VLA CASA Calibration Pipeline using CASA 6.4.1, which is designed to handle Stokes I continuum data, and is now available should you wish to access the calibrated data.
+
+{{sdm_id}}, observed on {{obs_end_time}}, {{bands}}, <filesize>
+
+These results have been checked by NRAO staff and notes about the quality assurance are below.
+
+{{#custom_text}}{{custom_text}}{{/custom_text}}{{^custom_text}}The results of this data processing request passed our quality assurance, with the following notes:{{/custom_text}}
+
+The following notes are also in the weblog:
+
+{{#workflow_metadata}}{{#qa}}{{qa}}{{/qa}}{{^qa}}No QA notes available.{{/qa}}{{/workflow_metadata}}{{^workflow_metadata}}No QA notes available.{{/workflow_metadata}}
+
+These products are not included in NRAO\'s Science Ready Data Products (SRDP) program but have been checked by NRAO staff for general quality.
+
+Some data may need further flagging before imaging: please check your data and the target calibration carefully before using the output from the pipeline for science!
+
+If your observations used a resolved calibrator source, but does not have standard model images in CASA, the pipeline calibration would not be accurate. In such instances, re-calibration using UV limits, or imaging the calibrator first and using the resulting image model, will be needed.
+
+If your science involves spectral lines, you should be aware of the following:
+
+1) The pipeline applies Hanning-smoothing by default, which may make the calibrated data set not optimal for certain spectral-line science.
+
+2) During the calibration process, several edge channels in each sub-band get flagged by default because they are noisier. Therefore, breaks in the frequency span get introduced in the pipeline calibrated data, which in turn may make the output not suitable for certain spectral-line science.
+
+3) The pipeline runs an RFI flagging algorithm which should flag strong lines and may remove spectral lines of interest to your science.
+
+Accessing Pipeline Products:
+
+There are two main pipeline products you may be interested in: the calibrated Measurement Set (MS) or the smaller pipeline product tar file that includes the weblog, final calibration and flag tables, a restore script, and other small files related to the calibration.
+For the entire Calibrated MS and pipeline tar file: you will need to request a calibrated MS (CMS) from our new archive access tool at https://data.nrao.edu . To access the pipeline products, please follow these steps:
+
+Click the "Log in" link at the top right of this page and select the NRAO login, which should take you to a page where you will enter your my.nrao.edu login information.
+Once logged in, you should be able to access your projects, both public and proprietary by selecting <yourusername>\'s data: navigate to the desired project and click the "+" symbol at the left to expand the list of available SBs.
+Find the specific observation you want and if you would like to recreate the pipeline calibration, click the "Add to clipboard" button to the left of that observation.
+Click the "Download" button at the top of the project list.
+In the pop-up window that opens, leave the default options the same: this should have the "Choose download data format:" option to "Calibrated Measurement Set", and the "Restore previous CMS" option should be filled in with a tar file named something like project-code_YYYY_MM_DD_THH_MM_SS.SSS.tar. Note that these two options are unavailable when more than one archive file is added to the clipboard. If the calibrated measurement set is needed for multiple scheduling blocks they must be downloaded one at a time. Since this calibration was done with CASA , 6.2.1-7 leave the "CASA Version:" drop-down menu at this version. Now click the "Submit Request" button.
+Once ready, you should receive an email with download instructions. This restoration of the calibrated MS may take several hours or longer, depending on the specifics of your observation.
+=====
+For just the pipeline tar file (includes the weblog, final flag versions, final calibration tables, restore script, and other related small files)
+Click the "Log in" link at the top right of this page and select the NRAO login, which should take you to a page where you will enter your my.nrao.edu login information.
+Once logged in, you should be able to access your projects, both public and proprietary: navigate to the desired project and click the "+" symbol at the left to expand the list of available SBs.
+Find the specific observation you want and if you would like to recreate the pipeline calibration, click the icon in the "Cals" column near the right side of the returned list, which should bring up a pop-up window to download a tar file named something like project-code_YYYY_MM_DD_THH_MM_SS.SSS.tar, and click "Submit Request". Once ready, you should receive an email with download instructions.
+
+More Information:
+
+For more information about the pipeline, including access to the scripted pipeline, instructions for rerunning the pipeline, applying pipeline calibration to raw data, or modification to suit your particular science goals, please go here:
+https://science.nrao.edu/facilities/vla/data-processing/pipeline
+
+Please let us know if you have any questions or concerns through the VLA Pipeline department of the Helpdesk at:
+https://help.nrao.edu/{{/is_srdp}}{{/metadata}}{{/parameters}}{{/version}}