Calibration and image annihilation
This refactors the Annihilator class and splits it into a bunch of subclasses where each subclass defines:
- Area to clean (e.g. spool, cache)
- Workflow to clean (if applicable)
- Max age to keep
- Whether or not to flag the workflow as cleaned
- How to get a list of stale directories (e.g. from workflow server, all directories, from db query)
The script was beginning to be split into a lot of conditionals all over the place so I thought I would make the "annihilate" logic not care about everything outside of its list of directories to clean.
Oh and this adds annihilation for calibrations and images (and the images' corresponding calibrations). Calibrations will be cleaned if they aren't marked as SRDP in 14 days. Images will be cleaned along with their parent calibrations 14 days after they're ingested.
Merge request reports
Activity
assigned to @dnemergu
- Resolved by Charlotte Hausman
please point this at main
added 15 commits
- 1abdbd7f...714e4c4c - 5 earlier commits
- 42e89d23 - Not done yet, but made the annihilator kind of an interface where subclasses...
- 81492db6 - Moving spool annihilator to be last because it marks workflows as cleaned, not...
- 3238474f - Changed the staging annihilator to simply delete all directories older than the capo property
- c50f05cc - Made get_days_to_keep more generic like it was
- c64c62d2 - Stale calibration annihilator
- dbf02caf - Stale image annihilators
- 49b3e5c2 - Fixing cyclical import
- 971d7bf1 - Throw a shoe if area_to_clean isn't defined by a subclass
- 4abcbc26 - Fixing silly query mistakes
- a174873f - Added a comment about not considering paths from another profile as stale....
Toggle commit list- Resolved by Daniel Nemergut
- Resolved by Daniel Nemergut
- Resolved by Sam Kagan
- Resolved by Sam Kagan
- Resolved by Daniel Nemergut
added 5 commits
- 8e58cc82 - Added a dry-run flag that doesn't delete directories nor flag workflow as cleaned
- 092420de - Forgot to commit this enum value fix
- 10751ff5 - Slapping a distinct on the images' associated calibrations due to failed...
- bce3d3df - Adding new capo properties to docker.properties
- ab79e5ed - Adding dry-run to readme
Toggle commit list286 291 if self.path_valid(dir_path, path): 287 if not pathlib.Path(dir_path).exists(): 288 if area_to_clean == Areas.SPOOL.value: 292 if not Path(dir_path).exists(): 293 if self._flag_cleaned and not self._args.dry_run: 289 294 logger.info( 290 295 f"directory {dir_path} not found, setting 'cleaned' flag on workflow request..." 291 296 ) 292 297 self.set_cleaned(dir_path) 293 298 else: 294 logger.info(f"directory {dir_path} not found in {area_to_clean}, skipping...") 299 logger.info(f"directory {dir_path} not found in {self._area_to_clean}, skipping...") 295 300 else: 296 301 logger.info(f"found directory {directory}, annihilating...") 297 self.annihilate_directory(area_to_clean, directory) 302 self.annihilate_directory(directory) The first thing
annihilate_directory
does is redetermine 'path' which already exists higher up in this method. Since this is the only location thatannihilate_directory
is used, would it be better to feed the already determined path variable from above intoannihilate_directory
as a parameter?
- Resolved by Charlotte Hausman
- Resolved by Daniel Nemergut
added 1 commit
- 43ca9336 - Putting the connection instantiation in the try block
added 1 commit
- fe8f737f - Added psycopg2 and sqlalchemy to annihilator toml
added 1 commit
- 0e544d74 - Added calibration and image cleanup to the --all argument
mentioned in commit 7d1dc23e