Removing rawdata race conditions

This prevents a couple of race conditions when reusing the rawdata directory from a previous execution's spool.

First, the workflows now make hard links to the rawdata instead of moving it out of the previous execution's spool. This way, if the previous execution is still running, there will still be a rawdata directory for it to work with.

Second, the prev_processing_dir parameter only gets passed to the workflow when the previous execution has finished running the productfetcher. This prevents copying over a partially fetched rawdata directory.

assigned to @dnemergu

the two new checks might produce key errors if the 'processing_dir' or 'fetch_end_time' keys aren't present when checking for values

Bah, true. I should only need to check for the keys existence and not the values at all I think, as I don't really care about the values here.

changed this line in version 2 of the diff

resolved all threads

Hard links seem like a very elegant solution here!

added 1 commit

• 1728a40d - Checking metadata keys not values

Compare with previous version

approved this merge request

approved this merge request

mentioned in commit 9c743c3c

merged