Skip to content

WS-528: Fix HTCondor environment for `ingest`

Nathan Hertz requested to merge add-mysql-oracle-env-vars-to-ingest-workflow into main

ingest_cal submit file:

  • Added environment variables that link external libraries for MySQL and Oracle
  • Added 1GB memory request to avoid an out-of-memory error I was getting

ingest PEX:

  • Rebuilt ingest PEX on RHEL 7 machine and pushed it up to the dev sbin area
  • Temporarily changed hard-coded ingestion manifest file name from ingestion-manifest.json -> ingestion_manifest.json (will change back after a manifest generator fix uses the former naming scheme; it currently uses the latter naming scheme) Changed back and redeployed.

ingest-request.sh:

  • Changed capo call to use pycapo
  • Added apt install curl to workflow container Dockerfiles (ingest-request.sh uses it)

ingest_cal workflow testing:

  • Ran an ingest workflow on dev and got past the MySQL/Oracle library issues
  • Ran a fully successful ingest_cal workflow

Miscellaneous:

  • Removed dependency on db container for schema-dev docker-compose service
  • Added a few gripes

Completes ticket WS-571

Edited by Nathan Hertz

Merge request reports