Contents

REQUEST AN ACCOUNT

Please use the computing facility request form available by clicking here

SYSTEM OVERVIEW

Stratus is a 1080-core computing cluster available to ARM investigators and users for ARM data analysis and visualization. Stratus consists of a 30-node Cray cluster with a total of 7.68 GB DDR4 memory per core. Each dual socket node consist of two Intel Xeon E5-2697V4 processors (18 cores per processor, 36 cores per node). It provides 57.6 TB fast Solid State Drive (SSD) storage and 100 TB parallel Lustre filesystem storage. It uses a high bandwidth Mellanox InfiniBand network. The system has two (2) external login nodes. Two of the system nodes also provides NVIDIA Kepler3 K80 (GK210) GB GPUs.

Cluster Policies

Please review below policy documents:

• To review the Stratus Compute policy document click here.
• To review the Stratus Storage policy document click here.
• To review the Stratus Cyber security policy document click here.

ACCESSING STRATUS CLUSTER (Back to Top)

SSH Access:

The Stratus cluster is accessed using a secured shell (SSH) client, using the following two steps. XCAMS/UCAMS accounts are required to access Stratus.

Step 1: Login to the login Server:

Step 2: Login to the Stratus Cluster:

• ssh username@stratus.ornl.gov

DATA STORAGE RESOURCES (Back to Top)

The ADC Cluster provides an array of data storage platforms, each designed with a particular purpose in mind. Storage areas are broadly divided into two categories: those intended for user data and those intended for project data. Within each of the two categories, we provide different sub-areas, each with an intended purpose:

User Home (Back to Top)

Home directories for each user are NFS-mounted on all ADC Cluster systems and are intended to store long-term, frequently-accessed user data. User Home areas are not backed up. This file system does not generally provide the input/output (I/O) performance required by most compute jobs, and is not available to compute jobs on most systems. See the section “Data Retention, Purge, & Quota Summary” for more details on applicable quotas, backups, purge, and retention timeframes.

User Scratch (Back to Top)

Project members get an individual User Scratch directory; these reside in the high-capacity Lustre® file system on large, fast disk areas intended for global (parallel) access to temporary/scratch storage. Because of the scratch nature of the file system, it is not backed up and files are automatically purged on a regular basis. Files should not be retained in this file system for long, but rather should be migrated to HPSS Archive space as soon as the files are not actively being used. If a file system associated with your User Scratch directory is nearing capacity, the ADC Cluster Support may contact you to request that you reduce the size of your Member scratch directory. See the section “Data Retention, Purge, & Quota Summary” for more details on applicable quotas, backups, purge, and retention timeframes.

Project Share (Back to Top)

Individual Project Share directories reside in the high-capacity Lustre file system on large, fast disk areas intended for global (parallel) access to temporary/scratch storage. Because of the scratch nature of the file system, it is not backed up. If a file system associated with Project Share storage is nearing capacity, the ADC Cluster Support may contact the PI of the project to request that he or she reduce the size of the Project scratch directory. See the section “Data Retention, Purge, & Quota Summary” for more details on applicable quotas, backups, purge, and retention timeframes.

World Share (Back to Top)

Each project has a World Share directory that resides in the high-capacity Lustre file system on large, fast disk areas intended for global (parallel) access to temporary/scratch storage. Because of the scratch nature of the file system, it is not backed up. If a file system associated with World Share storage is nearing capacity, the ADC Cluster may contact the PI of the project to request that he or she reduce the size of the World Work directory. See the section “Data Retention, Purge, & Quota Summary” for more details on applicable quotas, backups, purge, and retention timeframes.

Local Scratch Storage (Back to Top)

A fast solid state (SSD) disk area intended for parallel access to temporary storage in the form of scratch directories. This area is local to the computational node. This directory is, for example, intended to hold temporary and intermediate output generated by a user’s job. This is a run time only file system which is created at the start of a batch job and is purged at the end of the job. Files should not be retained in this file system and should be migrated to Lustre scratch or archival storage before finishing the job.

Path for local scratch storage is available during job runtime via environment variable $localscratch. Variable $localscratch typically has the form "/localscratch/tmp.$USER.$PBS_JOBID.or-condo-pbs01" and is specific to the user and to the scheduled job.

Project Storage (WARP) (Back to Top)

A NFS area intended for temporary data storage for moving data off the Lustre file system. This area is local to the computational node. This directory is, for example, intended to hold temporary and intermediate output generated by a user’s job. This is a run time only file system which is created at the start of a batch job and is purged at the end of the job. Files should not be retained in this file system and should be migrated to Lustre scratch or archival storage before finishing the job.

Data Transfer (Back to Top)

Scripts

We have provided two different methods to get data into the stratus cluster. The datastream_search tool allows access to datastreams that are too big or too old to be stored in the data center's fast storage. The adc_xfer tool allows users access to all the resources available in /data/archive and /data/datastream. For more information, see the table below:

Script Usage

Before you have access to these tools you need to export MODULEPATH (see documentation below). Then load the module data_wrapper/1.0.0 . Now you can use the data transfer scripts like a normal Linux program.

Transfer data from HPSS to Stratus lustre:

The optional arguments and the descriptions are listed below:

Transfer data from ADC to Stratus lustre:

Positional Arguments:

The optional arguments and the descriptions are listed below:

Summary

The following table details quota, backup, purge, and retention information for each user-centric and project-centric storage area available at the ADC Cluster.

Data Storage Resources

Default Shell (Back to Top)

The default shell on the Stratus cluster is bash. However, Stratus supports the following shells:

• bash
• tcsh
• csh
• ksh

Using Modules (Back to Top)

The modules software package allows you to dynamically modify your user environment by using pre-written module files.

Each module file contains the information needed to configure the shell for an application. After the modules software package is initialized, the environment can be modified on a per-module basis using the module command, which interprets a module file. Typically, a module file instructs the module command to alter or set shell environment variables such as PATH or MANPATH. Module files can be shared by many users on a system, and users can have their own personal collection to supplement and/or replace the shared module files. As a user, you can add and remove module files from your current shell environment. The environment changes performed by a module file are viewed by using the module command. More information on modules are found by running man module.

To access ARM specific softwares in your environment, add following to your .bashrc

Summary of module commands (Back to Top)

Available Software (Back to Top)

Additional software may be installed within the cluster, as required. To check list of available software run command module avail

To see list of softwares loaded in your environment, run command module list

These are the ARM specific modules

Other available software are:

Requesting new software

To request a new software, please contact cluster support

Compiling codes (Back to Top)

C, Fortran examples

coming soon.

MPI Example

                      
 //hello_mpi.c

 #include 
 #include 

int main(int argc, char** argv) {
  // Initialize the MPI environment
  MPI_Init(NULL, NULL);
  // Get the number of processes
  int world_size;
  MPI_Comm_size(MPI_COMM_WORLD, &world_size);
  // Get the rank of the process
  int world_rank;
  MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
  // Get the name of the processor
  char processor_name[MPI_MAX_PROCESSOR_NAME];
  int name_len;
  MPI_Get_processor_name(processor_name, &name_len);
  // Print off a hello world message
  printf("Hello world from processor %s, rank %d  out of %d processors\n", processor_name, world_rank, world_size);
  // Finalize the MPI environment.
  MPI_Finalize();
  }
                      
                   

To compile mpi example, please load gcc and openmpi modules. After loading the modules, use gcc to compile hello_mpi.c. For example,

                      
module load gcc/5.3.0
module load openmpi/2.0.0
gcc $OPENMPI_INC -o mpi_test.o -c mpi_test.c
                      
                   

CUDA/GPU example

coming soon.

IDL example

coming soon.

Available Queues: (Back to Top)

Example Submission Scripts:

Burst queue

Standard CPU queue:

GPU queue:

ARM production queue:

ARM production queue (arm-prod) has been specifically designed for daily production jobs. 15 nodes on Stratus queue is placed in a reservation every day during period of 12:00AM - 6:00AM to provide dedicated and guaranteed resources for critical operational jobs.

Details of the standing reservations can be checked using command “showres”.

Scheduling batch jobs (Back to Top)

Batch scripts, or job submission scripts, are the mechanism by which a user submits and configures a job for execution. A batch script is simply a shell script which contains:

• Commands that can be interpreted by batch scheduling software (e.g. PBS)
• Commands that can be interpreted by a shell

The batch script is submitted to the batch scheduler where it is parsed. Based on the parsed data, the batch scheduler places the script in the scheduler queue as a batch job. Once the batch job makes its way through the queue, the script will be executed on a service node within the set of allocated computational resources.

Sections of a Batch Script - Batch scripts are parsed into the following three sections:

1. The Interpreter Line: The first line of a script can be used to specify the script’s interpreter. This line is optional. If not used, the submitter's default shell will be used. The line uses the "hash-bang-shell" syntax: #!/path/to/shell
2. The Scheduler Options Section: The batch scheduler options are preceded by #PBS, making them appear as comments to a shell. PBS will look for #PBS options in a batch script from the script’s first line through the first non-comment line. A comment line begins with #. #PBS options entered after the first non-comment line will not be read by PBS.
3. The Executable Commands Section: The shell commands follow the last #PBS option and represent the main content of the batch job. If any #PBS lines follow executable statements, they will be ignored as comments.

The execution section of a script will be interpreted by a shell and can contain multiple lines of executable invocations, shell commands, and comments. When the job's queue wait time is finished, commands within this section will be executed on a service node (sometimes called a "head node") from the set of the job's allocated resources. Under normal circumstances, the batch job will exit the queue after the last line of the script is executed.

An example batch script:

The following table summarizes frequently-used options to PBS:

To submit job to the queue:

To check status of the jobs in the queue:

PBS sets multiple environment variables at submission time. The following PBS variables are useful within batch scripts: