Updated 2023-02-24

Run openMP Programs on the Cluster

Overview

  • openMP is a useful tool to write parallel programs in C, C++, and fortran
  • Workflow is similar to running other programs on the cluster:
    • First write SBATCH Script, which defines resources and compiles C program with openMP
    • Submit to scheduler to run on compute nodes
    • Gather results
  • Example C program for this guide can be found here. The program simply prints out "Hello World" from every thread that is running
  • Example SBATCH script, which will be explained, can be found here
  • Move the files to your account on the cluster, or write them on the cluster to follow along

Walkthrough: Run an Example openMP Script

#!/bin/bash
#SBATCH -JopenMPTest
#SBATCH -A [Account] 
#SBATCH -N2 --ntasks-per-node=4
#SBATCH --mem-per-cpu=2G
#SBATCH -t1
#SBATCH -qinferno
#SBATCH -oReport-%j.out

cd $SLURM_SUBMIT_DIR
echo "Started on `/bin/hostname`"
module load gcc/10.3.0
export OMP_NUM_THREADS=8

gcc -fopenmp openMP_example.c -o openMP_example
./openMP_example
  • The #SBATCH lines are directives, covered in depth in the Using Slurm on Phoenix Guide
  • $SLURM_SUBMIT_DIR is simply a variable that represents the directory you submit the SBATCH script from.
  • Make sure any dependencies are in that same directory you told the cluster to enter
  • The second line prints the name of the node that job is running on to the output file
  • Then, the preffered version of the c compiler is loaded. Many versions of gcc and intel compilers are available on the cluster. To find available versions of a software, run: 
module spider
module spider <software name> #ex: module spider gcc would show available verisons of gcc
  • to load a software, include the module load <module name> line in the pbs script. You must load the software's module before you run the software in the SBATCH script.
  • Next, you MUST set OMP_NUM_THREADS to the number of threads you requested in the beginning (nodes x ppn), otherwise you might slow down the system. In this case, it would be 4 x 2 = 8.

Warning

You must set OMP_NUM_THREADS to the number of threads specified in the top of the PBS script (nodes x ppn)

  • Then, the C program is compiled with openMP
  • Finally, the program is run, and the output is automatically recorded in the out file (which will show up in the directory you submitted the script from)

Part 2: Submit Job and Check Status

  • Be sure to change to the directory that contains the SBATCH Script
  • Submit as normal, with qsub <sbatch scriptname.sbatch>. In this case sbatch openmp.sbatch
  • Check job status with squeue --job <jobID>, replacing with the jobid returned after running sbatch
  • You can delete the job with scancel <jobID> , replacing with the jobid returned after running sbatch

Path 3: Collecting Results

  • Any files created by the script show up in the folder wehre the script was ran (unless otherwise programmed)
  • the outpt file be found by typing ls and looking for the output file you named in the SBATCH script, in this case Report-<jobID>.out.
  • To see the contents of the out file, run:
cat <output file name> #ex: cat Report-<jobID>.out
  • output for exmaple openMP script should print out "hello world" 8 times, one every thread:
---------------------------------------
Begin Slurm Prolog: Dec-01-2022 01:21:33
Job ID:    130809
User ID:   svangala3
Account:   phx-pace-staff
Job name:  openMPTest
Partition: cpu-small
QOS:       inferno
---------------------------------------
Started on atl1-1-01-003-32-1.pace.gatech.edu
Hello World
Hello World
Hello World
Hello World
Hello World
Hello World
Hello World
Hello World
---------------------------------------
Begin Slurm Epilog: Dec-01-2022 01:21:35
Job ID:        130809
Array Job ID:  _4294967294
User ID:       svangala3
Account:       phx-pace-staff
Job name:      openMPTest
Resources:     cpu=8,mem=16G,node=2
Rsrc Used:     cput=00:00:24,vmem=17212K,walltime=00:00:03,mem=0,energy_used=0
Partition:     cpu-small
QOS:           inferno
Nodes:         atl1-1-01-003-32-[1-2]
---------------------------------------
  • To move output files off the cluster, see file transfer guide
  • Congratulations! you have succesfully run a parallel C script using openMP on the cluster