This is two lines of thought combined into one merge:

1. AMQP clients should either receive messages or send messages
2. Capability queues are based on a database-backed queue manager rather than keeping state in-memory

Most of the work relating to the first idea comes in refactoring the Router to not be a message sender. Many places in the code now either instantiate a MessageSender instead, or use both a Router and a MessageSender if they truly needed both functionalities. The previous implementation appears to have caused messages to arrive out of order because facilities like `wf_monitor` that only send messages were also trying to receive messages, and either not handling them at all or putting them into a buffer of some kind to be dropped on the floor when the process ended.

The work relating to the second idea changes the way that steps are processed in the capability service and eliminates the capability engine concept. Now when PrepareAndRunWorkflow steps are reached, the capability is simply moved into the Waiting state and the queue manager is signaled. Whenever the queue manager is awakened, it checks to see if any queues have slots available and requests waiting. If they do, the number of available slots are used to get requests and start executing them. When an execution exits the cluster, the queue manager is signaled again, so the process continues until all the jobs are processed. As a stability benefit, we check this on startup as well.

- Local cluster can now run download workflows
- wf_monitor's timeout functionality now works the way we expect it to

changing default spl to smaller image spl

Daniel Lyons authored 4 years ago and Nathan Hertz committed 4 years ago

- Datafetcher runs in container and retrieves data
- Delivery can deliver data to a directory within the container

Charlotte Hausman authored 4 years ago and Nathan Hertz committed 4 years ago

Messaging system is now running on a more robust foundation:
- `kombu` package for AMQP networking
- New Messenger and Router objects
- New message format
- Easy-to-use callback subscribing
- Can now get messages anywhere in the system easily and send replies back easily

All enabled tests are now true unit tests with no external dependencies; new `requirements.txt` file in `test/` that installs testing requirements (`pytest`, `pytest-mock`, etc.); and `run-tests.sh` now has an extra step that installs these

All enabled tests now pass. TODO: Refactor integration tests into unit tests; add more test coverage

system since I figure that's on its way out the door)

updated WorkflowEvent shape

wf_monitor to use new amqp_helpers setup; updated wf_monitor tests;
updated WorkflowEventSchema to accept a string in the timestamp field
instead of a datetime (we could probably change this, but this is how
it's set up for now)

originally going to use a singleton pattern but ran into issues with
that)

Hopefully final changes to setup.py files regarding package name changes; renamed a amqp_helpers dir to channels

Made amqp_helpers directory structure match the other subprojects. Made some small changes to monitor.py and test_monitor.py

Added __str__ and __repr__ methods to WorkflowMonitor; small updates to same methods in WorkflowEvent