WS-1769 VLA manifest creation

This sets any *pbcor.tt0.fits images to be the science products and assigns similarly named images as ancillaries to them. If no science product with a similar name exists for an image, it'll be assigned as an ancillary to the first one to not leave any images behind.

assigned to @dnemergu

Here's an output example (added some junk data to check edge cases):

{
    "parameters": {
        "telescope": "EVLA",
        "reingest": "false",
        "ngas_ingest": "true",
        "ingestion_path": "/lustre/aoc/cluster/pipeline/docker/workspaces/staging/tmp4dbkvftp",
        "calibrate": "false",
        "additional_metadata": "./aux_image_metadata.json"
    },
    "input_group": {
        "science_products": [
            {
                "locator": ""
            }
        ]
    },
    "output_group": {
        "science_products": [
            {
                "type": "fits_image",
                "filename": "16B-069.MJD57713.51329133102.J1522+3934_sci.K_band.cont.I.pbcor.tt0.fits",
                "ancillary_products": [
                    {
                        "type": "spectral_index",
                        "filename": "16B-069.MJD57713.51329133102.J1522+3934_sci.K_band.cont.I.alpha.error.fits"
                    },
                    {
                        "type": "spectral_index",
                        "filename": "16B-069.MJD57713.51329133102.J1522+3934_sci.K_band.cont.I.alpha.fits"
                    },
                    {
                        "type": "clean_mask",
                        "filename": "16B-069.MJD57713.51329133102.J1522+3934_sci.K_band.cont.I.mask.fits"
                    },
                    {
                        "type": "primary_beam",
                        "filename": "16B-069.MJD57713.51329133102.J1522+3934_sci.K_band.cont.I.pb.tt0.fits"
                    },
                    {
                        "type": "fits_image",
                        "filename": "16B-069.MJD57713.51329133102.J1522+3934_sci.K_band.cont.I.tt0.fits"
                    },
                    {
                        "type": "fits_image",
                        "filename": "16B-069.MJD57713.51329133102.J1522+3934_sci.K_band.cont.I.tt1.fits"
                    },
                    {
                        "type": "spectral_index",
                        "filename": "16B-069.MJD57713.51329133102.J1522+3934_sci.Q_band.cont.I.alpha.error.fits"
                    }
                ]
            },
            {
                "type": "fits_image",
                "filename": "16B-069.MJD57713.51329133102.J1522+3934_sci.P_band.cont.I.pbcor.tt0.fits"
            },
            {
                "type": "fits_image",
                "filename": "16B-069.MJD57713.51329133102.J1522+3934_sci.X_band.cont.I.pbcor.tt0.fits",
                "ancillary_products": [
                    {
                        "type": "spectral_index",
                        "filename": "16B-069.MJD57713.51329133102.J1522+3934_sci.X_band.cont.I.alpha.error.fits"
                    },
                    {
                        "type": "spectral_index",
                        "filename": "16B-069.MJD57713.51329133102.J1522+3934_sci.X_band.cont.I.alpha.fits"
                    },
                    {
                        "type": "clean_mask",
                        "filename": "16B-069.MJD57713.51329133102.J1522+3934_sci.X_band.cont.I.mask.fits"
                    },
                    {
                        "type": "primary_beam",
                        "filename": "16B-069.MJD57713.51329133102.J1522+3934_sci.X_band.cont.I.pb.tt0.fits"
                    },
                    {
                        "type": "fits_image",
                        "filename": "16B-069.MJD57713.51329133102.J1522+3934_sci.X_band.cont.I.tt0.fits"
                    },
                    {
                        "type": "fits_image",
                        "filename": "16B-069.MJD57713.51329133102.J1522+3934_sci.X_band.cont.I.tt1.fits"
                    }
                ]
            }
        ],
        "ancillary_products": [
            {
                "type": "pipeline_weblog",
                "filename": "weblog.tgz"
            },
            {
                "type": "pipeline_artifacts",
                "filename": "pipeline_artifacts_2023_05_26T14_40_01.tar"
            },
            {
                "type": "ingestion_artifacts",
                "filename": "ingestion_artifacts_2023_06_01T04_11_14.694.tar"
            }
        ]
    }
}

approved this merge request

approved this merge request

approved this merge request

mentioned in commit 8da83d3f

merged

I was nervous about this line because in some languages the keys of a dictionary are not returned in any particular order, but it turns out in Python, they are returned in insertion order. So this would deterministically get you the last key, which is what I think you want.

Thanks for double checking that, I did want a way to know we're done iterating through the keys by checking the last element. I think in older Python versions this wasn't the case?

Per the documentation, this was changed in Python 3.7, I think before that it was not in a defined order.