Skip to content
Snippets Groups Projects
Commit c64d75fc authored by Daniel Lyons's avatar Daniel Lyons
Browse files

working on the environment definitions

parent 324c4aab
No related branches found
No related tags found
No related merge requests found
......@@ -4,8 +4,8 @@ Detailed documentation is available [on our Confluence page](https://open-conflu
## Overview
The NRAO archive system allows people to download and do some reprocessing of
radio astronomical observations made using NRAO-affiliated instruments:
The NRAO archive system allows people to download raw data and advanced SRDP
radio astronomical observation data products made using NRAO-affiliated instruments:
[The Very Large Array (VLA)](https://science.nrao.edu/facilities/vla),
[the Very Long Baseline Array (VLBA)](https://science.nrao.edu/facilities/vlba),
[the Atacama Large Millimeter Array (ALMA)](http://www.almaobservatory.org/en/home/)
......@@ -13,57 +13,11 @@ as well as the [the Green Bank Telescope (GBT)](http://greenbankobservatory.org/
## Components
The archive system is a constellation of subsystems each performing a critical
task.
This system has several subsystems which have their own subsystems. The two primary systems are:
* **amygdala** is the core of the nervous system for messaging and making
decisions about those messages
* **archiveIface** is the web-based user interface for starting downloads and
reprocessing
* the **data-fetcher** is responsible for retrieving archive files from NGAS
* **deployment** is our system for putting this system online
* **logback-utils** and **logback-servlet-utils** are provide logging services
* **mail** is a templated mailing system
* **messaging** provides messaging services the other components rely on
* **Model** provides Java models for the entities in the system
* **archive-solr** provides Solr indexing services for fast lookups
* **NGRH-ALMA-10_8** is the request handler and provides users with insight into
what step their download or reprocessing request is on
* **opencadc** and **tap-server** provide [Virtual Observatory](http://ivoa.net/)
services
* **pipeline-manifest-lib** and **ppr-schema** generate and parse reprocessing
requests and their results
* **schema** is the database schema used by the archive system
* **pyat** is the Python interface to the archive as well as the ingestion
system
* **workflow-all** provides cluster-based workflows for downloads, imaging and
calibration
* The **archive** which stores data products
* The **workspace** system which allows users to retrieve data products and transform them into other products
## How are requests processed?
To give a quick view of how the system works, let's walk through a single request.
1. The user arrives at the **archiveIface** at [archive-new.nrao.edu](http://archive-new.nrao.edu)
wanting to fetch some data
2. The user searches for a particular observation, such as 13B-014.
Behind the scenes, the **archiveIface** makes a request to a Solr index,
built by **archive-solr**, to find observations for 13B-014, which it
then presents to the user.
3. The user selects a data set and chooses download and reprocessing options
provided by **archiveIface** and clicks either Download or Reprocess.
4. **archiveIface** sends the request to **NGRH-ALMA-10_8** (the request handler).
5. **NGRH-ALMA-10_8** sends a workflow-start message to **workflow-all**.
6. **workflow-all** runs a sequence of workflow steps:
1. **ppr-schema** is used to generate pipeline processing request (PPR) for
the user's request
2. **data-fetcher** is used in the cluster to obtain the user's data
3. Other workflow tasks and jobs are used to run CASA and obtain the results
4. Finally, a message is sent back to **NGRH-ALMA-10_8* with the results
7. **NGRH-ALMA-10_8** shows the user their download is complete and how to obtain
the files.
All of the work is coordinated using AMQP messaging (via **messaging**) and the
database (defined by **schema**).
The pieces are broken down into applications (both web-based and command-line), RESTful services and shared libraries.
More details can be had by looking at our Confluence documentation.
......@@ -10,4 +10,6 @@ dependencies:
- pandas=1.0
- psycopg2=2.8
- pyramid=1.10
-
\ No newline at end of file
- requests=2.23
- tqdm=4.46
- pendulum=2.1
\ No newline at end of file
pika==1.1.0
pycapo==0.2.1post1
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment