Run Gnu Parallel on the Cluster¶
- Gnu Parallel allows users to build and execute shell command lines from standard input in parallel
- This guide will cover how to incorporate gnu parallel into a PBS script
- Load gnu parallel with
module load gnuparallel/20180822. Once you load gnu parallel, you can use it as you normally would
See all available versions with
module avail gnuparallel
Walkthrough: Use Gnu Parallel on the Cluster¶
- This walkthough will apply parallel to a simple
- PBS Script: gnuParallel.pbs
Part 1: The PBS Script¶
#PBS -N gnuParallel #PBS -A [Account] #PBS -l nodes=1:ppn=2 #PBS -l pmem=2gb #PBS -l walltime=1:00 #PBS -q inferno #PBS -j oe #PBS -o gnuParallel.out module load gnuparallel/20180822 parallel -k echo ::: A B C ::: D E F
#PBSdirectives are standard, requesting just 1 minute of walltime and 1 node with 2 cores. More on
#PBSdirectives can be found in the PBS guide
parallel -k echo ::: A B C ::: D E F: Runs echo with gnu parallel.
-k: Keeps sequence of output same as the order of input
A B Cand
D E Fare inputs, echo will print lines with one letter from
A B Cand one letter from
D E F
Part 2: Submit Job and Check Status¶
- Make sure you're in the dir that contains the
- Submit as normal, with
qsub <pbs script name>. In this case
- Check job status with
qstat -u <gtusername3> -n, replacing "gtusername3" with your gt username.
- You can delete the job with
qdel 22182721, replacing the number with the jobid returned after running qsub
Part 3: Collecting Results¶
- In the directory where you submitted the
PBSscript, you should see a
gnuParallel.outfile, which contains the results of the job. Use
cat gnuParallel.outor open the file in a text editor to take a look.
gnuParallel.outshould look like this:
Job name: gnuParallel Queue: inferno End PBS Prologue Mon Oct 1 10:01:09 EDT 2018 --------------------------------------- A D A E A F B D B E B F C D C E C F --------------------------------------- Begin PBS Epilogue Mon Oct 1 10:01:12 EDT 2018 Job ID: 22576977.shared-sched.pace.gatech.edu
- After the result files are produced, you can move the files off the cluster, refer to the file transfer guide for help.
- Congratulations! You successfully used gnu parallel on the cluster.