Futures.org

Workspace Architecture: Future Directions

Introduction

This document is intended to keep track of several things that don’t quite fit with the other architecture documentation:

• Things that sound like requirements, but weren’t
• Requirements that were missed between architecting and preparing for the CDR
• Hard, wall-like objects that we appear to be rushing towards with this architecture

Missed Requirements

At this time, there are no requirements that are known to have been missed.

Areas Needing Additional Design

Reingestion

In the course of creating the work breakdown, it has become clear that there is a need for a deeper understanding of reingestion. There is a technical problem here, in that the product locator system prevents us from safely implementing reingestion as a simple delete followed by an ingestion. There are several different semantics that need to be understood here; perhaps an earlier ingestion may have been corrupted or incomplete somehow, or perhaps we are really ingesting an improved version of something but want to retain the old one for some reason. The requirements here need to be sussed out in more depth before it can really be designed and implemented, and for this reason it is a bit vague.

Build and deployment

As the SSA group is in the midst of reviewing our core processes in advance of implementing workspaces, we see a need to treat the build system and deployment system for the workspace as component-level work deserving of the same kind of attention as the code itself. Testing has been a major concern for our planning since early in the process, but the emphasis on build and deployment is new. A certain amount of research is expected to be needed to figure out the right process here. The late-breaking development here is simply raising the importance of these issues, with the expectation that more details will be coming soon.

Parameter Validation

At the moment there is no parameter validation phase in the system. The architectural assumption here is that the UI will do its own validation and prevent users from doing nonsensical things. The UI is likely to generate its own parameter validation service, but since it isn’t a first-class architectural entity at the moment, it won’t be available for these systems to utilize. It seems likely that we will want to promote this to a first-class entity so that it can be used by the capability service itself to validate requests as they come in, even from other archive and workspace systems.

A future feature that might motivate more design here would be parameters whose values are influenced by the chosen products themselves. Some CASA parameters have sensible values that vary with different data files, for instance. There’s nothing in the current design to locate these values or validate them, and that is something we will probably be asked to revisit in the future.

Non-Requirements

Capability typing

I find it useful to mentally think of the type signatures of workflows and capabilities, if they were just like ordinary programming language artifacts. In this regime, capabilities are clearly functions from products to products and workflows are clearly procedures. In term of Java, you could see workflows and capabilities as having types like:

void workflow();
Product capability(Product input);

This leaves some work for the future:

• How do we handle checking the types of capability inputs?

  You can’t image a calibration table or generate a calibration table from an image, for instance.

• How do we verify the type of the object input and other inputs?
• How do we handle multiple products, such as for downloads? There’s only one product slot.
• How do we handle capabilities that need more than one product, each with different semantics?

  For instance, calibrate this raw data with this calibration table?

CASA version-specific UI

It’s true that different configuration for different CASA versions is possible within one capability. However, there is nothing in the system to modify the UI depending on the CASA version, in principle.

In practice, the UI parameter components are so completely independent, you could put conditional logic in them based on the CASA version that is chosen, as long as the receiving side is able to handle it. So if you only put key/value into the parameter when the CASA version is X, you’d better only use that key inside the override template for CASA version X. This may not be super fun to debug though, so it may be better to pretend you cannot do this.

Self-healing

This is not currently in-scope. Ingestion is a workflow, because it does not begin with a product. Reingestion, however, can be a capability because it begins with a product and ends with a product (the same product). So self-healing by reingestion can come into it here.

Custom triggering

It was realized fairly late in the design process that we are assuming a fairly straightforward replacement of some hard-coded rules in the existing archive rules engine (amygdala) with some other hard-coded rules in the same location to address the workspace system instead. Handling this properly with some flexibility would be a good idea, but there did not seem to really be a motivating requirement.

Auto-follow-ons

What if I pick raw data and want an image? In the current design, I have to set up the calibration or restore of that raw data, then I can send a follow-on request for an image from that calibrated MS. There is no way to do both of these in a single go.

I anticipate that implementing this feature on the current design should not be super hard. But it isn’t in-scope at this time.

Pre-emption

In this design, time-critical projects are flagged as such, and a time-critical standard calibration will always be chosen to run before a non-time-critical standard calibration. This is because the capability queues are priority queues.

There is no pre-emption in this design. This means that the arrival of a time-critical calibration will not cause a running non-time-critical calibration to be stopped or cancelled. The time-critical calibration still has to wait for whatever processing is currently running to finish. It should be the next thing executed though—unless there are more than N time-critical calibrations ahead of it in the queue, where N is the concurrency limit for this capability.

Cross-queue priorities

This design does not address priorities across different capabilities. There is no way, for instance, to specify that standard calibrations should be run preferentially over AUDI requests. There simply isn’t anything above the capability queues to make decisions like this; each queue will happily launch up to its concurrency limit of workflows.

However, because our wrapping for HTCondor is very flexible, we can probably fake this effect with HTCondor even though it isn’t surfaced in the architecture. In the definition of the workflow templates for HTCondor, we can add labels and conditions which HTCondor can be configured to use to create the effect of cross-queue priorities.