Updated 2023-03-31
Run openMP Programs on the Cluster¶
Overview¶
- openMP is a useful tool to write parallel programs in C, C++, and fortran
- Workflow is similar to running other programs on the cluster:
- First write SBATCH Script, which defines resources and compiles C program with openMP
- Submit to scheduler to run on compute nodes
- Gather results
- Example
C program
for this guide can be found here. The program simply prints out "Hello World" from every thread that is running - Example
SBATCH
script, which will be explained, can be found here - Move the files to your account on the cluster, or write them on the cluster to follow along
Walkthrough: Run an Example openMP Script¶
#!/bin/bash
#SBATCH -JopenMPTest
#SBATCH -A [Account]
#SBATCH -N2 --ntasks-per-node=4
#SBATCH --mem-per-cpu=2G
#SBATCH -t1
#SBATCH -qinferno
#SBATCH -oReport-%j.out
cd $SLURM_SUBMIT_DIR
echo "Started on `/bin/hostname`"
module load gcc/10.3.0
export OMP_NUM_THREADS=8
gcc -fopenmp openMP_example.c -o openMP_example
./openMP_example
- The #SBATCH lines are directives, covered in depth in the Using Slurm on Phoenix Guide
$SLURM_SUBMIT_DIR
is simply a variable that represents the directory you submit the SBATCH script from.- Make sure any dependencies are in that same directory you told the cluster to enter
- The second line prints the name of the node that job is running on to the output file
- Then, the preffered version of the c compiler is loaded. Many versions of gcc and intel compilers are available on the cluster. To find available versions of a software, run:
module spider
module spider <software name> #ex: module spider gcc would show available verisons of gcc
- to load a software, include the
module load <module name>
line in the pbs script. You must load the software's module before you run the software in the SBATCH script. - Next, you MUST set OMP_NUM_THREADS to the number of threads you requested in the beginning (nodes x ppn), otherwise you might slow down the system. In this case, it would be 4 x 2 = 8.
Warning
You must set OMP_NUM_THREADS to the number of threads specified in the top of the PBS script (nodes x ppn)
- Then, the C program is compiled with openMP
- Finally, the program is run, and the output is automatically recorded in the out file (which will show up in the directory you submitted the script from)
Part 2: Submit Job and Check Status¶
- Be sure to change to the directory that contains the
SBATCH
Script - Submit as normal, with qsub
<sbatch scriptname.sbatch>
. In this casesbatch openmp.sbatch
- Check job status with
squeue --job <jobID>
, replacing with the jobid returned after running sbatch - You can delete the job with
scancel <jobID>
, replacing with the jobid returned after running sbatch
Path 3: Collecting Results¶
- Any files created by the script show up in the folder wehre the script was ran (unless otherwise programmed)
- the outpt file be found by typing
ls
and looking for the output file you named in the SBATCH script, in this caseReport-<jobID>.out
. - To see the contents of the out file, run:
cat <output file name> #ex: cat Report-<jobID>.out
- output for exmaple openMP script should print out "hello world" 8 times, one every thread:
---------------------------------------
Begin Slurm Prolog: Dec-01-2022 01:21:33
Job ID: 130809
User ID: svangala3
Account: phx-pace-staff
Job name: openMPTest
Partition: cpu-small
QOS: inferno
---------------------------------------
Started on atl1-1-01-003-32-1.pace.gatech.edu
Hello World
Hello World
Hello World
Hello World
Hello World
Hello World
Hello World
Hello World
---------------------------------------
Begin Slurm Epilog: Dec-01-2022 01:21:35
Job ID: 130809
Array Job ID: _4294967294
User ID: svangala3
Account: phx-pace-staff
Job name: openMPTest
Resources: cpu=8,mem=16G,node=2
Rsrc Used: cput=00:00:24,vmem=17212K,walltime=00:00:03,mem=0,energy_used=0
Partition: cpu-small
QOS: inferno
Nodes: atl1-1-01-003-32-[1-2]
---------------------------------------
- To move output files off the cluster, see file transfer guide
- Congratulations! you have succesfully run a parallel C script using openMP on the cluster