Use the ARM Live Data Web Service to download data. You can query and automate machine-to-machine downloads of ARM data with this application.
Use a REST URL to enter:
saveData
or query
)Data files matching the criteria will be returned to the user. If a valid REST URL is entered, a page containing download links will be returned. You may also request that data file names be returned as an array in a JSON object in plain text by appending the wt=json parameter at the end of the URL.
To eliminate the manual step of following a link in an email to download data, use the ARM Live Data Web Service to set up cron jobs and automatically download data from /data/archive into your work space. All other data files not available in real-time on the server will have to go through the regular ordering process.
See script examples on how to bulk download ARM data.
Users may access ARM data via a REST URL. The format for the URL depends on the service type. Users may choose a service type of either
query
or saveData
. If query
is used, the REST URL may be entered in a browser, and a collection
of data download links will be returned. If saveData
is used, the REST URL may be used with either Wget or cURL to download
individual data files.
The tables below describe the parameters that must be used with both varieties of REST URLs. The capitolized words are placeholders for the values to be filled.
query
Format:
https://adc.arm.gov/armlive/data/query?user=USER_ID:ACCESS_TOKEN&ds=DATASTREAM&start=START_DATE&end=END_DATE
|
user |
|
ds |
|
start |
|
end |
|
wt=json
|
|
saveData
Format:
https://adc.arm.gov/armlive/data/saveData?user=USER_ID:ACCESS_TOKEN&file=DATA_FILE
|
user |
|
ds |
|
Users may access ARM data via several methods.
By default, entering a valid REST URL in a browser will return a page containing download links to data files.
The user may include the wt=json
parameter to instead return a JSON object
in plain text containing an array of the data file names, which can be used in scripts.
Both varieties use a service type of query
, an example user ID of example
,
an example access token of abcd1234
, request the datastream sgpmetE11.b1
,
and have start and end dates of January 1, 2018 and February 1, 2018, respectively.
Example URL, returns collection of download links. |
https://adc.arm.gov/armlive/data/query?user=example:abcd1234&ds=sgpmetE11.b1&start=2018-01-01T04:20:14.579Z&end=2018-02-01T04:20:14.579Z
|
Example URL with wt=json parameter, returns JSON object in plain text.
|
https://adc.arm.gov/armlive/data/query?user=example:abcd1234&ds=sgpmetE11.b1&start=2018-01-01&end=2018-02-01&wt=json
|
Wget and cURL are simple-to-use command line tools for downloading files. Wget downloads an individual data file. cURL returns data to the terminal, but the output can be piped into a file.
Below are examples of using both Wget and cURL to download the sgpmetE11.b1.20180101.000000.cdf
data file.
Using wget to retrieve data
|
$ wget 'https://adc.arm.gov/armlive/data/saveData?user=example:abcd1234&file=sgpmetE11.b1.20180101.000000.cdf'
|
Using curl to retrieve data
|
$ curl 'https://adc.arm.gov/armlive/data/saveData?user=example:abcd1234&file=sgpmetE11.b1.20180101.000000.cdf'
|
Note that the file downloaded with wget
using a service type of saveData
will not have the same name
as the original data file. To specify the filename, use the -O
option as demonstrated below:
Using the -O option with wget to specify downloaded file name
|
$ wget 'https://adc.arm.gov/armlive/data/saveData?user=example:abcd1234&file=sgpmetE11.b1.20180101.000000.cdf' -O sgpmetE11.b1.20180101.000000.cdf
|
Users may wish to perform batch downloads of data files instead of downloading one file at a time. This can be accomplished via scripts, such as the examples shown below.
armlive_getfiles
is a pip installable package repository that allows users to perform bulk downloads of ARM data.
If a user with an ID of example and an access token of abcd1234 wishes to download all data files from the sgpmetE11.b1 datastream recorded between January 1, 2018 and February 1, 2018, they would run the getFiles.py Python script with the appropriate arguments as shown below:
Installation: |
$ pip install git+https://code.ornl.gov/ofg/armlive_getfiles.git
|
Execution: |
$ getARMFiles -u example:abcd1234 -ds sgpmetE11.b1 -s 2018-01-01 -e 2018-02-01
|
OR
Installation: |
$ git clone https://code.ornl.gov/ofg/armlive_getfiles.git
|
Execution: |
$ python armlive_getfiles/src/getFiles.py -u example:abcd1234 -ds sgpmetE11.b1 -s 2018-01-01 -e 2018-02-01
|
You must be logged in to download scripts.
getFiles.sh
is a BASH script that allows users to perform bulk downloads of ARM data.
Last updated: 2018-07-25
If a user with an ID of example
and an access token of abcd1234
wishes to
download all data files from the sgpmetE11.b1
datastream recorded between January 1, 2018 and
February 1, 2018, they would run the getFiles.sh
BASH script with the appropriate arguments as shown below:
Usage of getFiles.sh
|
$ bash getFiles.sh example:abcd1234 sgpmetE11.b1 2018-01-01 2018-02-01
|
You must be logged in to download scripts.
DownloadgetFiles.sh
Raw data can be accessed via the ADRS Web Service (ADRSWS). Instructions on how to use ADRSWS and how to request an access token can be found below.
Registered users of ARM data, who have been provided with access keys, may use ADRSWS to order ARM data outside of the ARM Data Discovery Tool. ADRSWS provides the following functionality: