Skip to content
Snippets Groups Projects

Calibration and image annihilation

Merged Daniel Nemergut requested to merge ws1353-annihilator_images_cleanup into main
1 unresolved thread

This refactors the Annihilator class and splits it into a bunch of subclasses where each subclass defines:

  • Area to clean (e.g. spool, cache)
  • Workflow to clean (if applicable)
  • Max age to keep
  • Whether or not to flag the workflow as cleaned
  • How to get a list of stale directories (e.g. from workflow server, all directories, from db query)

The script was beginning to be split into a lot of conditionals all over the place so I thought I would make the "annihilate" logic not care about everything outside of its list of directories to clean.

Oh and this adds annihilation for calibrations and images (and the images' corresponding calibrations). Calibrations will be cleaned if they aren't marked as SRDP in 14 days. Images will be cleaned along with their parent calibrations 14 days after they're ingested.

Edited by Daniel Nemergut

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • added 1 commit

    Compare with previous version

  • Charlotte Hausman resolved all threads

    resolved all threads

  • Charlotte Hausman
  • Charlotte Hausman
  • Sam Kagan
  • Sam Kagan
  • Daniel Nemergut added 5 commits

    added 5 commits

    • 8e58cc82 - Added a dry-run flag that doesn't delete directories nor flag workflow as cleaned
    • 092420de - Forgot to commit this enum value fix
    • 10751ff5 - Slapping a distinct on the images' associated calibrations due to failed...
    • bce3d3df - Adding new capo properties to docker.properties
    • ab79e5ed - Adding dry-run to readme

    Compare with previous version

  • added 1 commit

    • cf73ea1c - Added dateutils to pyproject.toml

    Compare with previous version

  • Also added a --dry-run flag which will print the directory path to be deleted instead of performing the deletion and flagging.

  • added 1 commit

    Compare with previous version

  • added 1 commit

    Compare with previous version

  • 286 291 if self.path_valid(dir_path, path):
    287 if not pathlib.Path(dir_path).exists():
    288 if area_to_clean == Areas.SPOOL.value:
    292 if not Path(dir_path).exists():
    293 if self._flag_cleaned and not self._args.dry_run:
    289 294 logger.info(
    290 295 f"directory {dir_path} not found, setting 'cleaned' flag on workflow request..."
    291 296 )
    292 297 self.set_cleaned(dir_path)
    293 298 else:
    294 logger.info(f"directory {dir_path} not found in {area_to_clean}, skipping...")
    299 logger.info(f"directory {dir_path} not found in {self._area_to_clean}, skipping...")
    295 300 else:
    296 301 logger.info(f"found directory {directory}, annihilating...")
    297 self.annihilate_directory(area_to_clean, directory)
    302 self.annihilate_directory(directory)
    • The first thing annihilate_directory does is redetermine 'path' which already exists higher up in this method. Since this is the only location that annihilate_directory is used, would it be better to feed the already determined path variable from above into annihilate_directory as a parameter?

    • Please register or sign in to reply
  • Daniel Nemergut added 2 commits

    added 2 commits

    • ac61c7ea - Renaming restore imaging enum to allow restores to use the name later, also...
    • b20b16a0 - Fixed associated calibrations query to ensure we don't clean any extra calibrations

    Compare with previous version

  • Charlotte Hausman approved this merge request

    approved this merge request

  • added 1 commit

    • 43ca9336 - Putting the connection instantiation in the try block

    Compare with previous version

  • added 1 commit

    • fe8f737f - Added psycopg2 and sqlalchemy to annihilator toml

    Compare with previous version

  • added 1 commit

    Compare with previous version

  • added 1 commit

    • 0e544d74 - Added calibration and image cleanup to the --all argument

    Compare with previous version

  • Sam Kagan approved this merge request

    approved this merge request

  • mentioned in commit 7d1dc23e

  • Please register or sign in to reply
    Loading