Skip to content
Snippets Groups Projects

WS-1013: Create the pims_split Workspaces workflow

Merged WS-1013: Create the pims_split Workspaces workflow
pims-split-foo into main
2 unresolved threads
Merged Nathan Bockisch requested to merge pims-split-foo into main
2 unresolved threads

A first draft of the new pims_split workflow, based on input from our in-person working group

Edited by Nathan Bockisch

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Charlotte Hausman
  • Charlotte Hausman
  • Charlotte Hausman
  • Charlotte Hausman
  • Nathan Bockisch resolved all threads

    resolved all threads

  • Nathan Bockisch added 1 commit

    added 1 commit

    Compare with previous version

  • Charlotte Hausman
    Charlotte Hausman @chausman started a thread on an old version of the diff
    • shared/workspaces/alembic/versions/e8e6d54d8444_add_pimscache_split_workflow.py 0 → 100644
    60
    61 queue
    62
    63 """
    64
    65 restore_sh = """#!/bin/sh
    66 set -o errexit
    67
    68 {{^existingRestore}}
    69 cd rawdata/
    70 ../productfetcher --product-locator $1 --product-locator $2
    71 cd ../
    72 ./casa_envoy --restore -c metadata.json PPR.xml
    73 {{/existingRestore}}
    74 {{#existingRestore}}
    75 ./null $*
  • Charlotte Hausman
    Charlotte Hausman @chausman started a thread on the diff
    • shared/workspaces/alembic/versions/e8e6d54d8444_add_pimscache_split_workflow.py 0 → 100644
    71 cd ../
    72 ./casa_envoy --restore -c metadata.json PPR.xml
    73 {{/existingRestore}}
    74 {{#existingRestore}}
    75 ./null $*
    76 {{/existingRestore}}
    77 """
    78
    79 # Workflow run on each split
    80 split_workflow_condor = """executable = split_workflow.sh
    81 arguments = "$(split_dir)"
    82
    83 SPLIT_DIR = "$(split_dir)"
    84 SBIN_PATH = /lustre/aoc/cluster/pipeline/$ENV(CAPO_PROFILE)/workspaces/sbin
    85 VLASS_DIR = {{data_location}}
    86 SPOOL_DIR = {{spool_dir}}
  • Nathan Bockisch added 1 commit

    added 1 commit

    • 8900acde - Hopefully fixed DAG, null call, and redundant VLASS_DIR variable

    Compare with previous version

  • Nathan Bockisch added 1 commit

    added 1 commit

    Compare with previous version

  • Charlotte Hausman approved this merge request

    approved this merge request

  • Nathan Bockisch marked this merge request as ready

    marked this merge request as ready

  • Nathan Bockisch added 7 commits

    added 7 commits

    • 3fdd34b0...c7074643 - 2 commits from branch main
    • bc0b7267 - Added script sketch for review
    • 99d0984b - Updated split workflow with sketches of all stages
    • 7ae6d591 - Added fixes from review
    • 200fb5a3 - Hopefully fixed DAG, null call, and redundant VLASS_DIR variable
    • ed5eb5a4 - Fixed spacing issue in DAG

    Compare with previous version

    Toggle commit list
  • Nathan Bockisch enabled an automatic merge when the pipeline for ed5eb5a4 succeeds

    enabled an automatic merge when the pipeline for ed5eb5a4 succeeds

  • Nathan Bockisch merged

    merged

    • Please register or sign in to reply