Updated 2019-02-14

Run Gnu Parallel on the Cluster

Overview

  • Gnu Parallel allows users to build and execute shell command lines from standard input in parallel
  • This guide will cover how to incorporate gnu parallel into a PBS script
  • Load gnu parallel with module load gnuparallel/20180822. Once you load gnu parallel, you can use it as you normally would

Tip

See all available versions with module avail gnuparallel

Walkthrough: Use Gnu Parallel on the Cluster

  • This walkthough will apply parallel to a simple echo command.
  • PBS Script: gnuParallel.pbs

Part 1: The PBS Script

#PBS -N gnuParallel
#PBS -l nodes=1:ppn=2
#PBS -l pmem=2gb
#PBS -l walltime=1:00
#PBS -q force-6
#PBS -j oe
#PBS -o gnuParallel.out

module load gnuparallel/20180822
parallel -k echo ::: A B C ::: D E F

  • The #PBS directives are standard, requesting just 1 minute of walltime and 1 node with 2 cores. More on #PBS directives can be found in the PBS guide
  • parallel -k echo ::: A B C ::: D E F : Runs echo with gnu parallel.
    • -k: Keeps sequence of output same as the order of input
    • A B C and D E F are inputs, echo will print lines with one letter from A B C and one letter from D E F

Part 2: Submit Job and Check Status

  • Make sure you're in the dir that contains the PBS Script
  • Submit as normal, with qsub <pbs script name>. In this case qsub java.pbs
  • Check job status with qstat -u <gtusername3> -n, replacing "gtusername3" with your gt username.
  • You can delete the job with qdel 22182721 , replacing the number with the jobid returned after running qsub

Part 3: Collecting Results

  • In the directory where you submitted the PBS script, you should see a gnuParallel.out file, which contains the results of the job. Use cat gnuParallel.out or open the file in a text editor to take a look.
  • gnuParallel.out should look like this:
Job name:   gnuParallel
Queue:      force-6
End PBS Prologue Mon Oct  1 10:01:09 EDT 2018
---------------------------------------
A D
A E
A F
B D
B E
B F
C D
C E
C F
---------------------------------------
Begin PBS Epilogue Mon Oct  1 10:01:12 EDT 2018
Job ID:     22576977.shared-sched.pace.gatech.edu

  • After the result files are produced, you can move the files off the cluster, refer to the file transfer guide for help.
  • Congratulations! You successfully used gnu parallel on the cluster.