Skip to content
GitLab
Explore
Sign in
Register
Primary navigation
Search or go to…
Project
workspaces
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Wiki
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Build
Pipelines
Jobs
Pipeline schedules
Artifacts
Deploy
Releases
Package registry
Model registry
Operate
Environments
Terraform modules
Monitor
Incidents
Analyze
Value stream analytics
Contributor analytics
CI/CD analytics
Repository analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Terms and privacy
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
ssa
workspaces
Commits
c64d75fc
Commit
c64d75fc
authored
4 years ago
by
Daniel Lyons
Browse files
Options
Downloads
Patches
Plain Diff
working on the environment definitions
parent
324c4aab
No related branches found
No related tags found
No related merge requests found
Changes
3
Hide whitespace changes
Inline
Side-by-side
Showing
3 changed files
README.md
+6
-52
6 additions, 52 deletions
README.md
environment.yml
+3
-1
3 additions, 1 deletion
environment.yml
requirements.txt
+2
-0
2 additions, 0 deletions
requirements.txt
with
11 additions
and
53 deletions
README.md
+
6
−
52
View file @
c64d75fc
...
...
@@ -4,8 +4,8 @@ Detailed documentation is available [on our Confluence page](https://open-conflu
## Overview
The NRAO archive system allows people to download
and do some reprocessing of
radio astronomical observations made using NRAO-affiliated instruments:
The NRAO archive system allows people to download
raw data and advanced SRDP
radio astronomical observation
data product
s made using NRAO-affiliated instruments:
[
The Very Large Array (VLA)
](
https://science.nrao.edu/facilities/vla
)
,
[
the Very Long Baseline Array (VLBA)
](
https://science.nrao.edu/facilities/vlba
)
,
[
the Atacama Large Millimeter Array (ALMA)
](
http://www.almaobservatory.org/en/home/
)
...
...
@@ -13,57 +13,11 @@ as well as the [the Green Bank Telescope (GBT)](http://greenbankobservatory.org/
## Components
The archive system is a constellation of subsystems each performing a critical
task.
This system has several subsystems which have their own subsystems. The two primary systems are:
*
**amygdala**
is the core of the nervous system for messaging and making
decisions about those messages
*
**archiveIface**
is the web-based user interface for starting downloads and
reprocessing
*
the
**data-fetcher**
is responsible for retrieving archive files from NGAS
*
**deployment**
is our system for putting this system online
*
**logback-utils**
and
**logback-servlet-utils**
are provide logging services
*
**mail**
is a templated mailing system
*
**messaging**
provides messaging services the other components rely on
*
**Model**
provides Java models for the entities in the system
*
**archive-solr**
provides Solr indexing services for fast lookups
*
**NGRH-ALMA-10_8**
is the request handler and provides users with insight into
what step their download or reprocessing request is on
*
**opencadc**
and
**tap-server**
provide
[
Virtual Observatory
](
http://ivoa.net/
)
services
*
**pipeline-manifest-lib**
and
**ppr-schema**
generate and parse reprocessing
requests and their results
*
**schema**
is the database schema used by the archive system
*
**pyat**
is the Python interface to the archive as well as the ingestion
system
*
**workflow-all**
provides cluster-based workflows for downloads, imaging and
calibration
*
The
**archive**
which stores data products
*
The
**workspace**
system which allows users to retrieve data products and transform them into other products
## How are requests processed?
To give a quick view of how the system works, let's walk through a single request.
1.
The user arrives at the
**archiveIface**
at
[
archive-new.nrao.edu
](
http://archive-new.nrao.edu
)
wanting to fetch some data
2.
The user searches for a particular observation, such as 13B-014.
Behind the scenes, the **archiveIface** makes a request to a Solr index,
built by **archive-solr**, to find observations for 13B-014, which it
then presents to the user.
3.
The user selects a data set and chooses download and reprocessing options
provided by
**archiveIface**
and clicks either Download or Reprocess.
4.
**archiveIface**
sends the request to
**NGRH-ALMA-10_8**
(the request handler).
5.
**NGRH-ALMA-10_8**
sends a workflow-start message to
**workflow-all**
.
6.
**workflow-all**
runs a sequence of workflow steps:
1.
**ppr-schema**
is used to generate pipeline processing request (PPR) for
the user's request
2.
**data-fetcher**
is used in the cluster to obtain the user's data
3.
Other workflow tasks and jobs are used to run CASA and obtain the results
4.
Finally, a message is sent back to
**
NGRH-ALMA-10_8
*
with the results
7.
**NGRH-ALMA-10_8**
shows the user their download is complete and how to obtain
the files.
All of the work is coordinated using AMQP messaging (via
**messaging**
) and the
database (defined by
**schema**
).
The pieces are broken down into applications (both web-based and command-line), RESTful services and shared libraries.
More details can be had by looking at our Confluence documentation.
This diff is collapsed.
Click to expand it.
environment.yml
+
3
−
1
View file @
c64d75fc
...
...
@@ -10,4 +10,6 @@ dependencies:
-
pandas=1.0
-
psycopg2=2.8
-
pyramid=1.10
-
\ No newline at end of file
-
requests=2.23
-
tqdm=4.46
-
pendulum=2.1
\ No newline at end of file
This diff is collapsed.
Click to expand it.
requirements.txt
0 → 100644
+
2
−
0
View file @
c64d75fc
pika
==1.1.0
pycapo
==0.2.1post1
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment