Skip to content

No Story: Ingest manifest marshmallow

Daniel Nemergut requested to merge ingest_manifest_marshmallow into development

This switches ingest_envoy to use marshmallow schemas to generate the ingestion manifest JSON files. This works by defining a schema class for each manifest component (e.g. AncillaryProductSchema for AncillaryProduct objects). The schema classes define the fields they will output by specifying attributes like the key (optional), type, required, nullable, etc. Field validation then happens when schema.dumps() is called to output the JSON. Note that validation doesn't normally happen for marshmallow's dumping, only loading objects from JSON. I had to add a hook to validate our dumps.

Some of our previous validations now live in the schema classes but some of them still lives on the objects (e.g. setting destination_dir = staging_source_dir).

Merge request reports

Loading