This is two lines of thought combined into one merge:

1. AMQP clients should either receive messages or send messages
2. Capability queues are based on a database-backed queue manager rather than keeping state in-memory

Most of the work relating to the first idea comes in refactoring the Router to not be a message sender. Many places in the code now either instantiate a MessageSender instead, or use both a Router and a MessageSender if they truly needed both functionalities. The previous implementation appears to have caused messages to arrive out of order because facilities like `wf_monitor` that only send messages were also trying to receive messages, and either not handling them at all or putting them into a buffer of some kind to be dropped on the floor when the process ended.

The work relating to the second idea changes the way that steps are processed in the capability service and eliminates the capability engine concept. Now when PrepareAndRunWorkflow steps are reached, the capability is simply moved into the Waiting state and the queue manager is signaled. Whenever the queue manager is awakened, it checks to see if any queues have slots available and requests waiting. If they do, the number of available slots are used to get requests and start executing them. When an execution exits the cluster, the queue manager is signaled again, so the process continues until all the jobs are processed. As a stability benefit, we check this on startup as well.

Janet Goldstein authored 3 years ago and Nathan Hertz committed 3 years ago

WS-254, WS-253, WS-251: Catch ingest-complete event, parse it, and create request based on given info

old name    new name
---         ---
ready       created
executing   submitted
?           in_progress

This reverts commit 942beee0

removed EXPOSE cmd in frontend's Dockerfile.dev and added empty dependencies to CI stages that don't need artifacts for efficiency

- Capability request status page now grabs all correct information from its request and fills that into the UI
- If the request is complete and the execution has a delivery URL, a download button appears on the page linking to the download area

- Two new `npm` packages (not including deps): `eslint` (linter for TypeScript code) and `prettier` (multi-language web-stack code formatter)
- New workspaces component: `capability-request` which is where the work for the page is being done

WS-195, WS-76: Notification System now working and sending emails on request submission and completion

Daniel Lyons authored 4 years ago and Nathan Hertz committed 4 years ago

Added the ability to compare capability executions by their creation timestamp; made it so encountering exceptions when receiving messages does not block the system

Charlotte Hausman authored 4 years ago and Nathan Hertz committed 4 years ago

Messaging system is now running on a more robust foundation:
- `kombu` package for AMQP networking
- New Messenger and Router objects
- New message format
- Easy-to-use callback subscribing
- Can now get messages anywhere in the system easily and send replies back easily

Updated `testing/utils/conftest.py` with reliable mock data that can be used in tests; added `shared/workspaces` tests

Nathan Hertz authored 4 years ago and Nathan Hertz committed 4 years ago

existing executions cannot be deleted (they must be disabled instead,
which will have to be implemented)

and capability request)

Refactored execution management logic out of `CapabilityService` into a new class `ExecutionManager`

- Added PATH_PREFIX variable to build stage to handle dockerfiles at paths other than /services
- removed .unit-test template from .gitlab-ci.yml because it was duplicated

`CapabilityService.create_request` instead of duplicating that logic
within the view; changed tests and mocks to go along with the changes