WS-1966 Mark4 import workflow

There are 2 alembic scripts here:

1. Modifies the ingest_mk_four_obs workflow to rename mark4 directories when it's done to indicate the ingestion status.
2. Makes the mk_four_import workflow which simply runs the mark4_import script from /users/vlapipe/workflows/PROFILE/bin/mark4_import.

While looking at the ingest_mk_four workflow I realized that I forgot to set requires_lustre to true for the ingest_obs workflow. I set it manually on dev and test and it needed to be in the migration script so it'll be set on prod too.

shared/workspaces/alembic/versions/91091612b6d0_mark4_workflow_renaming.py

+"""mark4 workflow renaming
+
+Revision ID: 91091612b6d0
+Revises: 569416c40ca8
+Create Date: 2023-09-22 13:34:54.054114
+
+"""
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision = '91091612b6d0'
+down_revision = '569416c40ca8'
+branch_labels = None
+depends_on = None
+
+
+old_ingest_mk_four_obs_sh = """#!/bin/sh
+set -o errexit
+
+# Get NGAS hosts and set up variables to randomly select one
+NGASHOSTSTR=$(./pycapo -q archive-ingestion.NGASHosts)
+NGASHOSTARR=(`/bin/echo ${NGASHOSTSTR}`) # Put the space-delimited host list into an array
+NGASHOSTLEN=${#NGASHOSTARR[@]}
+
+# Copy from the difx area to the Workspaces staging area
+WSSTAGINGDIR=$(./pycapo -q edu.nrao.workspaces.IngestionSettings.stagingDirectory)
+/bin/cp -r $1 $WSSTAGINGDIR
+
+OBSDIR=$(/bin/basename $1)
+/bin/chmod -R 750 $WSSTAGINGDIR/$OBSDIR # Make sure NGAS has permissions to ingest the files
+cd $WSSTAGINGDIR/$OBSDIR
+
+for FILE in *; do
+    # Pick random NGAS host to distribute the ingestion load
+    NGASINDEX=$(($RANDOM % $NGASHOSTLEN))
+    NGASHOST=${NGASHOSTARR[$NGASINDEX]}
+
+    FULLPATH=$(/bin/readlink -f $FILE)
+    NGASCMD="${NGASHOST}ARCHIVE?filename=file://${FULLPATH}"
+
+    INGESTOUT=$(/bin/curl $NGASCMD)
+    if echo $INGESTOUT | grep -i "error"; then
+        echo "Failed to ingest ${FILE}"
+        exit 1
+    fi
+done"""
+
+# Workflow will rename the directory to indicate the ingestion status
+new_ingest_mk_four_obs_sh = """#!/bin/sh
+set -o errexit
+
+ingested=$1.ingested
+failed=$1.failed
+
+# Get NGAS hosts and set up variables to randomly select one
+NGASHOSTSTR=$(./pycapo -q archive-ingestion.NGASHosts)
+NGASHOSTARR=(`/bin/echo ${NGASHOSTSTR}`) # Put the space-delimited host list into an array
+NGASHOSTLEN=${#NGASHOSTARR[@]}
+
+# Copy from the difx area to the Workspaces staging area
+WSSTAGINGDIR=$(./pycapo -q edu.nrao.workspaces.IngestionSettings.stagingDirectory)
+/bin/cp -r $1 $WSSTAGINGDIR
+
+OBSDIR=$(/bin/basename $1)
+/bin/chmod -R 750 $WSSTAGINGDIR/$OBSDIR # Make sure NGAS has permissions to ingest the files
+cd $WSSTAGINGDIR/$OBSDIR
+
+for FILE in *; do
+    # Pick random NGAS host to distribute the ingestion load
+    NGASINDEX=$(($RANDOM % $NGASHOSTLEN))
+    NGASHOST=${NGASHOSTARR[$NGASINDEX]}
+
+    FULLPATH=$(/bin/readlink -f $FILE)
+    NGASCMD="${NGASHOST}ARCHIVE?filename=file://${FULLPATH}"
+
+    INGESTOUT=$(/bin/curl $NGASCMD)
+    if echo $INGESTOUT | grep -i "error"; then
+        echo "Failed to ingest ${FILE}"
+        mv $1 $failed
+        exit 1
+    fi
+done
+
+mv $1 $ingested
+"""
+
+
+def upgrade():
+    op.execute(
+        f"""
+        UPDATE workflow_templates
+        SET content=E'{new_ingest_mk_four_obs_sh}'
+        WHERE filename='ingest_mk_four_obs.sh'
+        """
+    )
+
+
+def downgrade():
+    op.execute(
+        f"""
+        UPDATE workflow_templates
+        SET content=E'{old_ingest_mk_four_obs_sh}'
+        WHERE filename='ingest_mk_four_obs.sh'
+        """
+    )