Updated 2021-05-17

Run NCO on the Cluster

Overview

  • netCDF Operators is a suite of programs designed to facilitate manipulation and analysis of self-describing data stored in netCDF format.
  • This guide will cover how to run NCO on the Cluster.
  • You can find NCO home page here.
  • You can find a useful slideshow about using NCO here.

Summary

  • NCO includes many different programs, so the exact commands you execute depend on what operation you are trying to perform on your netCDF file.
  • For this guide, we will show you how to run an NCO operation on the Cluster which you can substitute for any other operation that fits your needs.

Walkthrough: Run [Software] on the Cluster

  • This walkthrough will cover how to output longitude and latitude data from a netCDF file.
  • The example netCDF file can be downloaded here
  • PBS Script can be found here
  • You can transfer the files to your account on the cluster to follow along. The file transfer guide may be helpful.

Part 1: The PBS Script

#PBS -N ncoTest
#PBS -A [Account]
#PBS -l nodes=1:ppn=2
#PBS -l pmem=2gb
#PBS -l walltime=3:00
#PBS -q inferno
#PBS -j oe
#PBS -o ncoTest.out

cd $PBS_O_WORKDIR
module load intel/15.0
module load hdf5/1.8.14
module load netcdf/4.3.3
module load nco/4.6.0

ncks -v lat,lon sresa1b_ncar_ccsm3-example.nc
  • The #PBS directives are standard, requesting just 3 minutes of walltime and 1 node with 2 cores. More on #PBS directives can be found in the PBS guide
  • $PBS_O_WORKDIR is a variable that represents the directory you submit the PBS script from. Make sure the files you want to use are in the same directory you put the PBS script.
  • Output Files will also show up in this dir as well
  • module load nco/4.6.0 loads the 4.6.0 version of NCO. To see what NCO versions are available, run module avail NCO, and load the one you want. The other module are dependencies that must be loaded before NCO is loaded.
  • ncks -v lat,lon sresa1b_ncar_ccsm3-example.nc is used to output the lat and lon variables from sresa1b_ncar_ccsm3-example.nc.

Part 2: Submit Job and Check Status

  • Make sure you're in the dir that contains the PBS Script as well as the NCO program
  • Submit as normal, with qsub <pbs script name>. In this case qsub nco.pbs
  • Check job status with qstat -t 22182721, replacing the number with the job id returned after running qsub
  • You can delete the job with qdel 22182721 , again replacing the number with the jobid returned after running qsub

Part 3: Collecting Results

  • In the directory where you submitted the PBS script, you should see a ncoTest.out file which contains the lat and lon ouput from sresa1b_ncar_ccsm3-example.nc. Use cat or open the file in a text editor to take a look.
  • ncoTest.out should look like this
  • After the result files are produced, you can move the files off the cluster, refer to the file transfer guide for help.
  • Congratulations! You successfully ran NCO on the cluster.