Updated 2021-05-17
Run NCO on the Cluster¶
Overview¶
- netCDF Operators is a suite of programs designed to facilitate manipulation and analysis of self-describing data stored in netCDF format.
- This guide will cover how to run NCO on the Cluster.
- You can find NCO home page here.
- You can find a useful slideshow about using NCO here.
Summary¶
- NCO includes many different programs, so the exact commands you execute depend on what operation you are trying to perform on your netCDF file.
- For this guide, we will show you how to run an NCO operation on the Cluster which you can substitute for any other operation that fits your needs.
Walkthrough: Run [Software] on the Cluster¶
- This walkthrough will cover how to output longitude and latitude data from a netCDF file.
- The example netCDF file can be downloaded here
- PBS Script can be found here
- You can transfer the files to your account on the cluster to follow along. The file transfer guide may be helpful.
Part 1: The PBS Script¶
#PBS -N ncoTest
#PBS -A [Account]
#PBS -l nodes=1:ppn=2
#PBS -l pmem=2gb
#PBS -l walltime=3:00
#PBS -q inferno
#PBS -j oe
#PBS -o ncoTest.out
cd $PBS_O_WORKDIR
module load intel/15.0
module load hdf5/1.8.14
module load netcdf/4.3.3
module load nco/4.6.0
ncks -v lat,lon sresa1b_ncar_ccsm3-example.nc
- The
#PBS
directives are standard, requesting just 3 minutes of walltime and 1 node with 2 cores. More on#PBS
directives can be found in the PBS guide $PBS_O_WORKDIR
is a variable that represents the directory you submit the PBS script from. Make sure the files you want to use are in the same directory you put the PBS script.- Output Files will also show up in this dir as well
module load nco/4.6.0
loads the 4.6.0 version of NCO. To see what NCO versions are available, runmodule avail NCO
, and load the one you want. The other module are dependencies that must be loaded before NCO is loaded.ncks -v lat,lon sresa1b_ncar_ccsm3-example.nc
is used to output the lat and lon variables fromsresa1b_ncar_ccsm3-example.nc
.
Part 2: Submit Job and Check Status¶
- Make sure you're in the dir that contains the
PBS
Script as well as theNCO
program - Submit as normal, with
qsub <pbs script name>
. In this caseqsub nco.pbs
- Check job status with
qstat -t 22182721
, replacing the number with the job id returned after running qsub - You can delete the job with
qdel 22182721
, again replacing the number with the jobid returned after running qsub
Part 3: Collecting Results¶
- In the directory where you submitted the
PBS
script, you should see ancoTest.out
file which contains the lat and lon ouput fromsresa1b_ncar_ccsm3-example.nc
. Usecat
or open the file in a text editor to take a look. ncoTest.out
should look like this- After the result files are produced, you can move the files off the cluster, refer to the file transfer guide for help.
- Congratulations! You successfully ran NCO on the cluster.