Updated 2021-05-17
Run CDO on the Cluster¶
Overview¶
- Climate Data Operators is a collection of command-line operators to manipulate and analyze climate and numerical weather prediction data.
- This guide will cover how to run CDO on the Cluster.
- More information about CDO can be found on their homepage.
Summary¶
- There are various operators for you to use that can be viewed with
cdo -h
after loading the necessary modules. - For this walkthrough, we will cover basic operations such as displaying variables, timesteps, and information about the underlying grid and file conversion.
Walkthrough: Run CDO on the Cluster¶
- This walkthrough will go through basic usage of CDO as outlined in their Tutorial Page.
sst.mon.mean.nc
can be found here- PBS Script can be found here
- You can transfer the files to your account on the cluster to follow along. The file transfer guide may be helpful.
Part 1: The PBS Script¶
#PBS -N cdoTest
#PBS -A [Account]
#PBS -l nodes=1:ppn=2
#PBS -l pmem=4gb
#PBS -l walltime=3:00
#PBS -q inferno
#PBS -j oe
#PBS -o cdoTest.out
cd $PBS_O_WORKDIR
module load intel/15.0
module load cdo/1.9.5
# Display variables of a file
cdo -showname sst.mon.mean.nc
# Display number of timesteps of a file
cdo -ntime sst.mon.mean.nc
# Display Information about the underlying grid
cdo -griddes sst.mon.mean.nc
# File conversion with different file types: Copying whole data sets can be easily done with the copy operator.
# With the '-f' switch, you can choose a new file type:
cdo -f grb -copy sst.mon.mean.nc sst.mon.mean.grb
- The
#PBS
directives are standard, requesting 3 minutes of walltime and 1 node with 2 cores. More on#PBS
directives can be found in the PBS guide $PBS_O_WORKDIR
is a variable that represents the directory you submit the PBS script from. Make sure the files you want to use are in the same directory you put the PBS script.- Output Files will also show up in this dir as well
module load nco/1.9.5
loads the 1.9.5 version of CDO. To see what version of a software are available, runmodule avail [Software]
, and load the one you want. The other module is a dependency that must be loaded before CDO is loaded.- The comment above each line in the PBS Script above explains what that line does.
Part 2: Submit Job and Check Status¶
- Make sure you're in the dir that contains the
PBS
Script as well as theCDO
program - Submit as normal, with
qsub <pbs script name>
. In this caseqsub cdo.pbs
- Check job status with
qstat -t 22182721
, replacing the number with the job id returned after running qsub - You can delete the job with
qdel 22182721
, again replacing the number with the jobid returned after running qsub
Part 3: Collecting Results¶
- In the directory where you submitted the
PBS
script, you should see acdoTest.out
file which contains the results of the job andsst.mon.mean.grb
that was generated as an example of file conversion. Usecat
or open the file in a text editor to take a look. cdoTest.out
should look like this:
---------------------------------------
Begin PBS Prologue Mon Jul 15 13:17:37 EDT 2019
Job ID: 26373952.shared-sched.pace.gatech.edu
User ID: svemuri8
Job name: cdoTest
Queue: inferno
End PBS Prologue Mon Jul 15 13:17:37 EDT 2019
---------------------------------------
cdo showname: Processed 1 variable [0.01s 12MB]
sst
cdo ntime: Processed 1 variable over 1542 timesteps [0.05s 13MB]
1542
cdo griddes: Processed 1 variable [0.01s 12MB]
#
# gridID 1
#
gridtype = lonlat
gridsize = 64800
xsize = 360
ysize = 180
xname = lon
xlongname = "Longitude"
xunits = "degrees_east"
yname = lat
ylongname = "Latitude"
yunits = "degrees_north"
xfirst = 0.5
xinc = 1
yfirst = 89.5
yinc = -1
cdo copy: Processed 99921600 values from 1 variable over 1542 timesteps [5.04s 21MB]
---------------------------------------
Begin PBS Epilogue Mon Jul 15 13:17:59 EDT 2019
Job ID: 26373952.shared-sched.pace.gatech.edu
User ID: svemuri8
Job name: cdoTest
Resources: neednodes=1:ppn=2,nodes=1:ppn=2,pmem=4gb,walltime=00:03:00
Rsrc Used: cput=00:00:00,energy_used=0,mem=0kb,vmem=0kb,walltime=00:00:22
Queue: inferno
Nodes:
iw-c42-25.pace.gatech.edu
End PBS Epilogue Mon Jul 15 13:17:59 EDT 2019
---------------------------------------
- You should also see a
sst.mon.mean.grb
file that is over 100 MB large in the working directory. - After the result files are produced, you can move the files off the cluster, refer to the file transfer guide for help.
- Congratulations! You successfully ran CDO on the cluster.