envoys.rst

Envoy Architecture

What is an Envoy?

An Envoy is defined by the Merriam-Webster dictionary as “a person delegated to represent one government in its dealings with another”.

In the context of the Workspaces System, an Envoy is a python executable program that is delegated the task of interacting with a system external to Workspaces from within a workflow. This interaction includes making sure all required inputs are present and correct as well and launching the external system in question.

The envoys that currently exist are the :doc:`CARTA envoy <../tools/carta_envoy>`, :doc:`CASA envoy <../tools/casa_envoy>` and :doc:`ingest envoy <../tools/ingest_envoy>`.

When should I build a new Envoy?

If you are needing to interact with a system external to workspaces, i.e. a system which is not built and maintained within the workspaces project, you should create an envoy. Examples of external systems Workspaces interacts with are CASA or the ingest system.

How do I build a new Envoy?

An Envoy is essentially a bridge between Workspaces and the system of interest. As such, a Workspaces Envoy has two essential functions: Setup and Launch.

Basic Envoy Architecture

An envoy is expected to be extensible if required, i.e. it should be able to launch multiple types of calls to the system of interest. As such, envoys should typically have the following structure: - A main entry point which determines which type of call to make based on supplied parser input, this file is typically named after the envoy (unless it’s the main file for CASA Envoy which is aptly named “palaver”) - A number of typed launcher classes which handle the type specific setup (typically broken up into other classes) and then make the call to execute the external system. These classes are typically contained in a file called launchers.py but can also rely on any number of other classes for setup functionality.

Example: CASA Envoy has two main modules palaver.py, the entry point, and launchers.py, which contains the CalibrationLauncher and ImageLauncher classes. However, the package also contains the setup helper modules auditor.py and foundation.py, which audit the input files for required fields and correct for HTCondor submission, and ensure all data is in the required locations for CASA execution respectively.

What should the setup phase cover?

This is the most complex part of any envoy and typically encapsulates multiple stages.

Because the external system already exists, we cannot dictate the inputs provided for execution nor the environment needed for it to run successfully. Therefore, we need to gather all required information and ensure it’s in place before making the call to run the system.

Steps that might occur in the setup phase include: - setting environment variables required by the system being launched - generating required input files, such as tar files - ensuring file placement in specific locations - ensuring required files contain all required information - ensuring required files are corrected for HTCondor processing if necessary

What should the launch phase cover?

Once setup is complete all requirements to call the system should be satisfied. The only thing that should happen in the launch phase is the execution call to the system of interest.

Currently, launching an external system has been found to have little variation when there are multiple types of action that the system is being asked to perform. Example, CASA calibration and CASA imaging have the same inputs - PPR.xml - and therefore only one way of launching CASA has been needed. However, it is possible that a system might need to be launched with different inputs which might require different launchers.

What happens after the external system exits?

This step can be viewed as optional as not all envoys might need it. Sometimes there are post execution steps that should be completed within the envoy before it exits as they might impact the completion status.

Example: CASA likes to pretend everything is gloriously fine when it exits and doesn’t throw error codes. Therefore, the envoy is required to check the casa log after CASA exits to make sure it really really really did complete successfully.