ways to run regression tests
Option 1 (the current bamboo plan)
runner:pytest -> test collected
-> test1/eval
-> test2/eval
-> test3/eval
-> test4/eval
results assembled by runner:pytest -> pipeline_regression_test_result.xml
- switching the working directory under a single session could be problematic (e.g. plotms)
- long-running time and suboptimal computing resource usage (even on a single node)
Option 2
runner:pytest-xdist -> test collected
-> runner:worker1: test1:pp/eval -> test2:pp/eval
-> runner:worker2: test3/eval -> test4/eval
results assembled by runner:pytest: pipeline_regression_test_result.xml
- switching the working directory under a single session could be problematic (e.g. plotms)
- still long running time and needs careful planning to avoid saturating runner
Option 3
runner:pytest-xdist -> test collected
-> "runner:worker1": test1:node1:casa-session1/runner:eval -> test2:node2:casa-session1/runner:eval
-> "runner:worker2": test3:node3:casa-session3/runner:eval -> test4:node4:casa-session4/runner:eval
results assembled by runner:pytest: pipeline_regression_test_result.xml
+ works in terms of performing regression test analysis
+ eval is still handled by runner
+ from the runner point of view, still one command + one xml output
+ each casa-session can have its own setup.
- pytest warning capturing (warning summary) can't reach warnings inside individual casa-sessions;
- trackback may be confusing if something breaks inside the casa-session handling ppr/recipe calls.
Option 4
runner:scheduler:select (daily/weekly/hifa/hifv/hsd/..) -> test subset
-> node1:pytest::test1-call -> test1_results.xml
-> node2:pytest::test2-call -> test2_results.xml
-> node3:pytest::test3-call -> test3_results.xml
results assembed by runner:scheduler
- Can Bamboo display multiple xml as the test suite progressing
- a Python code (scheduler) to do scheduling and post-processing xml stitched into: pipeline_regression_test_result.xml? then Bamboo displays it.
Note
- the communication between the runner and cluster nodes can be done with
plutils.helper.queue
-
plutils.helper.pcasa.casa_session
could be used to perform the pipeline processing workflow under independent CASA sessions with customized configuration (mpicasa vs. casa, lproc, etc.) - see the Bamboo plan log (https://open-bamboo.nrao.edu/browse/PIPE-TPM-90/log) for the current workflow (option 1)
Edited by Rui Xue