Updated 2021-05-17

Run openMP Programs on the Cluster

Overview

  • openMP is a useful tool to write parallel programs in C, C++, and fortran
  • Workflow is similar to running other programs on the cluster:
    • First write PBS Script, which defines resources and compiles C program with openMP
    • Submit to scheduler to run on compute nodes
    • Gather results
  • Example C program for this guide can be found here. The program simply prints out "Hello World" from every thread that is running
  • Example PBS script, which will be explained, can be found here
  • Move the files to your account on the cluster, or write them on the cluster to follow along

Walkthrough: Run an Example openMP Script

Part 1: The PBS Script & Loading the Compiler

#PBS -N openMP_example_script
#PBS -l nodes=2:ppn=4
#PBS -l pmem=2gb
#PBS -l walltime=00:30
#PBS -q inferno
#PBS -j oe
#PBS -o openMP_example.out
#PBS -m abe
#PBS -M shollister7@gatech.edu


cd ~/test_directory
echo "Started on `/bin/hostname`"
module load gcc/4.9.0
export OMP_NUM_THREADS=8

gcc -fopenmp openMP_example.c -o openMP_example
./openMP_example
  • The #PBS lines are directives, covered in depth in the PBS guide
  • The first step is to tell the cluster to enter the directory where the parallel C program is located (in this case ~/test_directory, a dir I have made in my home directory). Should be whatever directory you put the C script to be run in.
  • Make sure any dependencies are in that same directory you told the cluster to enter
  • The second line prints the name of the node that job is running on to the output file
  • Then, the preffered version of the c compiler is loaded. Many versions of gcc and intel compilers are available on the cluster. To find available versions of a software, run
module avail
module avail <software name> #ex: module avail gcc would show available verisons of gcc
  • to load a software, include the module load <module name> line in the pbs script. You must load the software's module before you run the software in the pbs script.
  • Next, you MUST set OMP_NUM_THREADS to the number of threads you requested in the beginning (nodes x ppn), otherwise you might slow down the system. In this case, it would be 4 x 2 = 8.

Warning

You must set OMP_NUM_THREADS to the number of threads specified in the top of the PBS script (nodes x ppn)

  • Then, the C program is compiled with openMP
  • Finally, the program is run, and the output is automatically recorded in the out file (which will show up in the directory you submitted the script from)

Part 2: Submit the PBS Script and Collect Results

  • make sure your're in the folder where the openMP PBS Script is located, and run *
qsub <scriptName.pbs>  #ex: qsub openMP.pbs for this guide
  • if successful, this will print something like 2180446.shared-sched-pace.gatech.edu
  • the number in the beginning is the job id, useful for checking predicted wait time in queue or job status
  • After a couple seconds, find estimated wait time in queue with
showstart <jobID>
  • check job status with
qstat <jobID> or qstat -u someuser3 -n
  • for more ways to check status, how to cancel job, and more useful commands, checkout the command cheatsheet
  • Any files created by the script will show up in the folder where the script was ran (unless otherwise programmed)
  • the output file will be found by typing ls and looking for the output file you named in the PBS script, in this case openMP_example.out
  • To see the contents of the out file, run
cat <output file name>  #ex: cat openMP_example.out
  • output for example openMP script should print out "hello world" 8 times, one for every thread:

Screenshot

  • To move output files off the cluster, see file transfer guide
  • Congratulations! you have succesfully run a parallel C script using openMP on the cluster