WS-3239 EVLA SDM ingest_obs
These are the changes to ingest_envoy necessary for EVLA SDM ingestion. It gets from receiving the workflow request via curl to calling ingest. No changes to the workflow service and wrester ended up being necessary.
Main changes:
- Converted the bash observation product collector to Python.
- Modified the product collector to only use hard links if the source and staging directories are on the same device, otherwise it'll perform a copy (to prevent cross-device errors).
- Cleaned up path vs string usage in the collectors.
To test:
- Put an SDM directory somewhere with some XML files in it.
- Submit an
ingest_obsworkflow request pointing to the SDM directory. Here's what I did (essentially what StackStorm will do):
import requests
url = "http://workflow:3456/workflows/ingest_obs/requests/create-and-submit"
params = {
"telescope": "EVLA",
"data_src": "/lustre/mchammer/evla/sdm/123-ABC_sb30512915_1.57112.35963144676",
"project_code": "123-ABC",
}
response = requests.post(url, json=params)
- View results. The source directory will be renamed to
*.runningand then*.failedor*.ingested, the staging directory should be named*.runningand theingest_obsspool will contain the logs.
Edited by Daniel Nemergut