Delivery rework
7 unresolved threads
7 unresolved threads
Clean up destination code and add some functionality
- Split the destinations into separate files
- New DestinationTempFile for writing tempfiles into the destination which eventually get added
- All of the close() methods are now streaming
- Implemented ChecksumDecorator, so we get a SHA1SUMS file
- Implemented FetchFile decorator, so we get a rudimentary fetch-all.sh script
- Configured Docker and delivery for local development to have a separate volume mount for serving files
Merge request reports
Activity
19 20 # generate the hash and keep it 21 self.sumsfile.file().writelines([self.hash_file_line(file, relative_path).encode("utf8")]) 22 23 def hash_file_line(self, file: pathlib.Path, relative_path: str): 24 """ 25 Generate a line for the checksum file. 26 27 :param file: file to hash 28 :param relative_path: relative path to show in checksum file 29 :return: a line like "<hash> <relative_path>" 30 """ 31 return f"{self.hash_file(file)} {relative_path}\n" 32 33 @staticmethod 34 def hash_file(file: pathlib.Path): 1 import pathlib 2 3 from .interfaces import DestinationDecorator, Destination 4 5 6 class FetchFileGenerator(DestinationDecorator): 7 """ 8 Generates a shell script which can be used to fetch all the files related to a request. 9 The script uses "wget". 10 """ 11 12 def __init__(self, underlying: Destination): -
[nm; I was looking in wrong place]
Edited by Janet Goldstein
63 kinds of destination (globus or streaming or something like that) we will need to greatly reconsider how to 64 implement this. 65 """ 66 67 def __init__( 68 self, 69 destination: LocalDestination, 70 relative_path: str, 71 tempfile: tempfile.NamedTemporaryFile, 72 ): 73 self.destination = destination 74 self.relative_path = relative_path 75 self.tempfile = tempfile 76 77 def close(self): 78 # The key idea here is that after we close the temp file but before we delete it, we add it to the destination 138 # 139 ## the following is reported to be a memory efficient way to get md5 checksum 140 ## https://stackoverflow.com/questions/3431825/generating-an-md5-checksum-of-a-file > post by Omnifarious 141 # def hash_bytestr_iter(bytesiter, hasher): 142 # for block in bytesiter: 143 # hasher.update(block) 144 # return hasher 145 # 146 # def file_as_blockiter(afile, blocksize=65536): 147 # with afile: 148 # block = afile.read(blocksize) 149 # while len(block) > 0: 150 # yield block 151 # block = afile.read(blocksize) 152 153 14 # class SubdirectoryDecorator(DestinationDecorator): changed this line in version 3 of the diff
22 compare_dirs = filecmp.dircmp( 23 temp_directory + "/" + eb_name, (test_data_path + eb_name) 24 ) 23 compare_dirs = filecmp.dircmp(temp_directory + "/" + eb_name, (test_data_path + eb_name)) 25 24 # did the comparison report they are the same 26 assert ( 27 len(compare_dirs.left_only) == 0 28 and len(compare_dirs.right_only) == 0 29 and len(compare_dirs.funny_files) == 0 30 ) 25 assert len(compare_dirs.left_only) == 0 26 assert len(compare_dirs.right_only) == 0 27 assert len(compare_dirs.funny_files) == 0 31 28 32 29 33 30 def test_local_rawdata_with_tar(tmpdir_factory): added 1 commit
- 74ebed80 - Document a few things, break out the subdirectory decorator even though it isn't used currently
added 7 commits
-
74ebed80...71b97f8f - 2 commits from branch
main - e11f3e7c - Considerable rework here:
- 73bd43c5 - Split destination classes out into separate files and finish cleaning up the tests
- bc60ae0c - Configure the local environment so that files wind up in /var/www and are served properly by nginx
- af746d56 - Clean up the tests per Janet's remarks
- 32ac7fd4 - Document a few things, break out the subdirectory decorator even though it isn't used currently
Toggle commit list-
74ebed80...71b97f8f - 2 commits from branch
57 128 class DestinationDecorator(Destination): 58 129 def __init__(self, underlying: Destination): 59 130 self.underlying = underlying 60 super().__init__(underlying.context, underlying.path) 131 super().__init__(underlying.context) 61 132 62 133 def add_file(self, file: pathlib.Path, relative_path: str): 63 134 self.underlying.add_file(file, relative_path) 64 135 65 def add_directory(self, directory: pathlib.Path, relative_path: str): 66 self.underlying.add_directory(directory, relative_path) 136 def create_file(self, relative_path: str) -> DestinationTempFile: -
the comment here (deliverer:137 )can be pydoc:
""" This is a bit of a hack, but there's only one thing that makes temporary files. If we don't have the following lines of code, adding the tempfile at the end of processing goes directly to the LocalDestination rather than passing through all the layers of the layer. :return: the temp file """
Please register or sign in to reply