Run CDO on the Cluster¶
- Climate Data Operators is a collection of command-line operators to manipulate and analyze climate and numerical weather prediction data.
- This guide will cover how to run CDO on the Cluster.
- More information about CDO can be found on their homepage.
- There are various operators for you to use that can be viewed with
cdo -hafter loading the necessary modules.
- For this walkthrough, we will cover basic operations such as displaying variables, timesteps, and information about the underlying grid and file conversion.
Walkthrough: Run CDO on the Cluster¶
- This walkthrough will go through basic usage of CDO as outlined in their Tutorial Page.
sst.mon.mean.nccan be found here
- PBS Script can be found here
- You can transfer the files to your account on the cluster to follow along. The file transfer guide may be helpful.
Part 1: The PBS Script¶
#PBS -N cdoTest #PBS -l nodes=1:ppn=2 #PBS -l pmem=4gb #PBS -l walltime=3:00 #PBS -q force-6 #PBS -j oe #PBS -o cdoTest.out cd $PBS_O_WORKDIR module load intel/15.0 module load cdo/1.9.5 # Display variables of a file cdo -showname sst.mon.mean.nc # Display number of timesteps of a file cdo -ntime sst.mon.mean.nc # Display Information about the underlying grid cdo -griddes sst.mon.mean.nc # File conversion with different file types: Copying whole data sets can be easily done with the copy operator. # With the '-f' switch, you can choose a new file type: cdo -f grb -copy sst.mon.mean.nc sst.mon.mean.grb
#PBSdirectives are standard, requesting 3 minutes of walltime and 1 node with 2 cores. More on
#PBSdirectives can be found in the PBS guide
$PBS_O_WORKDIRis a variable that represents the directory you submit the PBS script from. Make sure the files you want to use are in the same directory you put the PBS script.
- Output Files will also show up in this dir as well
module load nco/1.9.5loads the 1.9.5 version of CDO. To see what version of a software are available, run
module avail [Software], and load the one you want. The other module is a dependency that must be loaded before CDO is loaded.
- The comment above each line in the PBS Script above explains what that line does.
Part 2: Submit Job and Check Status¶
- Make sure you're in the dir that contains the
PBSScript as well as the
- Submit as normal, with
qsub <pbs script name>. In this case
- Check job status with
qstat -t 22182721, replacing the number with the job id returned after running qsub
- You can delete the job with
qdel 22182721, again replacing the number with the jobid returned after running qsub
Part 3: Collecting Results¶
- In the directory where you submitted the
PBSscript, you should see a
cdoTest.outfile which contains the results of the job and
sst.mon.mean.grbthat was generated as an example of file conversion. Use
cator open the file in a text editor to take a look.
cdoTest.outshould look like this:
--------------------------------------- Begin PBS Prologue Mon Jul 15 13:17:37 EDT 2019 Job ID: 26373952.shared-sched.pace.gatech.edu User ID: svemuri8 Job name: cdoTest Queue: force-6 End PBS Prologue Mon Jul 15 13:17:37 EDT 2019 --------------------------------------- cdo showname: Processed 1 variable [0.01s 12MB] sst cdo ntime: Processed 1 variable over 1542 timesteps [0.05s 13MB] 1542 cdo griddes: Processed 1 variable [0.01s 12MB] # # gridID 1 # gridtype = lonlat gridsize = 64800 xsize = 360 ysize = 180 xname = lon xlongname = "Longitude" xunits = "degrees_east" yname = lat ylongname = "Latitude" yunits = "degrees_north" xfirst = 0.5 xinc = 1 yfirst = 89.5 yinc = -1 cdo copy: Processed 99921600 values from 1 variable over 1542 timesteps [5.04s 21MB] --------------------------------------- Begin PBS Epilogue Mon Jul 15 13:17:59 EDT 2019 Job ID: 26373952.shared-sched.pace.gatech.edu User ID: svemuri8 Job name: cdoTest Resources: neednodes=1:ppn=2,nodes=1:ppn=2,pmem=4gb,walltime=00:03:00 Rsrc Used: cput=00:00:00,energy_used=0,mem=0kb,vmem=0kb,walltime=00:00:22 Queue: force-6 Nodes: iw-c42-25.pace.gatech.edu End PBS Epilogue Mon Jul 15 13:17:59 EDT 2019 ---------------------------------------
- You should also see a
sst.mon.mean.grbfile that is over 100 MB large in the working directory.
- After the result files are produced, you can move the files off the cluster, refer to the file transfer guide for help.
- Congratulations! You successfully ran CDO on the cluster.