WS-3239 EVLA SDM ingest_obs

These are the changes to ingest_envoy necessary for EVLA SDM ingestion. It gets from receiving the workflow request via curl to calling ingest. No changes to the workflow service and wrester ended up being necessary.

Main changes:

  • Converted the bash observation product collector to Python.
  • Modified the product collector to only use hard links if the source and staging directories are on the same device, otherwise it'll perform a copy (to prevent cross-device errors).
  • Cleaned up path vs string usage in the collectors.

To test:

  1. Put an SDM directory somewhere with some XML files in it.
  2. Submit an ingest_obs workflow request pointing to the SDM directory. Here's what I did (essentially what StackStorm will do):
import requests
url = "http://workflow:3456/workflows/ingest_obs/requests/create-and-submit"
params = {
    "telescope": "EVLA",
    "data_src": "/lustre/mchammer/evla/sdm/123-ABC_sb30512915_1.57112.35963144676",
    "project_code": "123-ABC",
}
response = requests.post(url, json=params)
  1. View results. The source directory will be renamed to *.running and then *.failed or *.ingested, the staging directory should be named *.running and the ingest_obs spool will contain the logs.
Edited by Daniel Nemergut

Merge request reports

Loading