Skip to content

Removing rawdata race conditions

Daniel Nemergut requested to merge ws2227-recycle_raw_data_improvements into main

This prevents a couple of race conditions when reusing the rawdata directory from a previous execution's spool.

First, the workflows now make hard links to the rawdata instead of moving it out of the previous execution's spool. This way, if the previous execution is still running, there will still be a rawdata directory for it to work with.

Second, the prev_processing_dir parameter only gets passed to the workflow when the previous execution has finished running the productfetcher. This prevents copying over a partially fetched rawdata directory.

Merge request reports

Loading