Web Service to Order ARM Data
Table of Contents
- Getting Started
- Restrictions
- Overview of the web service
- Convenience shell script
- Usage examples
- Help/Questions
Getting Started
To get started with the ADRS Web Service please do the following:
- Download the convenience shell script here
- Request an access key by emailing armarchive@ornl.gov
- Place the access key in ${HOME}/.adrsws-key and change the permissions to 600
- See Usage examples for detailed examples
Restrictions
This web service is designed for limited use by ARM users who need to order raw files or specific versions of files, other than the latest version.Overview of the web service
Registered users of ARM data, who have been provided with access keys, may use this web service to order ARM data outside of the ARM Data Discovery Tool.
The web service provides the following functionality:
- Return a list of all datastreams. Output is a JSON object.
- Given a datastream and date range, return a list of data filenames, as stored in the ARM systems. Output is a JSON object.
- Given a list of data filenames, place an order for data and return the Order ID and URL to track the state of order processing. Output is a JSON object.
The web service accepts a JSON object, via the POST method. The JSON object contains the various parameters that the web service needs to service a request.
- Details of web service:
-
URL: https://archive.arm.gov/cgi-bin/adrsws.py
Number of request parameters: 1
Request parameter name: reqjson
Request JSON object specification:
When reqtype == "dslist" Key Name Requirement Description accesskey Mandatory A unique key given to a user of this web service. Keys can be shared within a group. The purpose of this key to prevent DoS (Denial of Service) attacks and unauthorized placement of orders for ARM data. userid Mandatory The user ID of a registered ARM user in good standing. reqtype Mandatory Value should be: "dslist"
When "reqtype": "dates" Key Name Requirement Description accesskey Mandatory See above userid Mandatory See above reqtype Mandatory Value should be: "dates" datastream Mandatory Full data stream name. (e.g., sgp30baebbrE39.c1)
When "reqtype": "flist" Key Name Requirement Description accesskey Mandatory See above userid Mandatory See above reqtype Mandatory Value should be: "flist" withvers Optional Takes either "y" or "n". If "y", then the version extensions on the retrieved filenames are retained. datastream Mandatory Full data stream name. (e.g., sgp30baebbrE39.c1) startdate Mandatory Start date in the following format: YYYY-mm-dd enddate Mandatory End date in the following format: YYYY-mm-dd -
When reqtype == "order" Key Name Requirement Description accesskey Mandatory See above userid Mandatory See above reqtype Mandatory Value should be: "order" withvers Optional Takes either "y" or "n". If "y", then the version extensions on the retrieved files are retained. fileslist Mandatory A list containing the file names to be retrieved. If version extension is specified, then that exact version is retrieved. If no version extension is supplied, the latest version is returned. Duplicates in the list are automatically removed and invalid file names are ignored.
When reqtype == "userid" Key Name Requirement Description accesskey Mandatory See above reqtype Mandatory Value should be: "userid" email Mandatory Email address of the user as defined in the ARM database
Examples:
Request JSON where user "<your userid>" requests list of all datastreams:
{ "accesskey": "__ACCESS_KEY_FROM_ARM_ADC__", "userid": "<your userid>", "reqtype": "dslist" }
Request JSON where user "<your userid>" requests list of files from sgpmfrsraod1michE31.c1 datastream for a date range from 2016-01-01 to 2016-02-01, with version designators left intact on the filenames:
{ "accesskey": "__ACCESS_KEY_FROM_ARM_ADC__", "userid": "<your userid>", "withvers": "n", "reqtype": "flist", "datastream": "sgpmfrsraod1michE31.c1", "startdate": "2016-01-01", "enddate": "2016-02-01" }
Request JSON where user "<your userid>" orders 3 files:
{ "accesskey": "__ACCESS_KEY_FROM_ARM_ADC__", "userid": "<your userid>", "withvers": "n", "reqtype": "order", "fileslist": [ "sgpmfrsraod1michE31.c1.20160101.000000.cdf", "sgpmfrsraod1michE31.c1.20160102.000000.cdf", "sgpmfrsraod1michE31.c1.20160103.000000.cdf" ] }
Convenience shell script
A convenience shell script that will create the required JSON object and make the call to the web service is provided. It can be downloaded from:https://adc.arm.gov/docs/adrsws-sh.tgz [MD5(adrsws-sh.tgz)= 28502620442c3b10d5fdb60940130829]
Usage examples
To find out the ARM User ID of a user, given the user's email address:adrsws.sh -t userid -a <email address>To get a list of all datastreams as the above sample user, <your userid>:
adrsws.sh -t dslist -u <your userid>To get the start and end dates for a datastream:
adrsws.sh -t dates -u <your userid> -dTo get a list of files from the sgpmfrsraod1michE31.c1 datastream for a date range from 2016-01-01 to 2016-02-01, with version designators left intact on the filenames, again for our sample user:
adrsws.sh -u <your userid> -t flist -v -d sgpmfrsraod1michE31.c1 -s 2016-01-01 -e 2016-02-01To collect those file names in plain text format (one file name per line), instead of the default JSON format, in a file called "filelist.txt", use the "-p" flag. The command would then be:
adrsws.sh -u <your userid> -t flist -v -d sgpmfrsraod1michE31.c1 -s 2016-01-01 -e 2016-02-01 -p > filelist.txtTo order specific files defined in "filelist.txt", with version designators intact (ex. sgpmfrsraod1michE31.c1.20150101.000000.cdf.v1), the command would be:
adrsws.sh -u <your userid> -v filelist.txtor
cat filelist.txt | adrsws.sh -u <your userid> -v
Pipelining is also supported. For example, to get a list of file names and place the order in one consolidated step, the command would be:
adrsws.sh -u <your userid> -t flist -d sgpmfrsraod1michE31.c1 -s 2016-01-01 -e 2016-02-01 -p \ | adrsws.sh -u <your userid>
To have the ordered files automatically downloaded to a specific destination directory, say /home/<your userid>/arm-downloads, the command would be:
adrsws.sh -u <your userid> -t flist -d sgpmfrsraod1michE31.c1 -s 2016-01-01 -e 2016-02-01 -p \ | adrsws.sh -u <your userid> -g /home/<your userid>/arm-downloads
Running:
adrsws.shwithout any arguments, will display detailed help on usage.
OUTPUT INFORMATION
The output from the web service and convenience shell script will be JSON. The convenience shell script can take a "-p" flag to have the output in plain text. The "-p" flag only works when combined with "-t dslist" or "-t flist" flags.Help/Questions
To request an access key or for help/questions, please send email to:armarchive@ornl.gov