Updated 2021-05-17

Run SUNDIALS on the Cluster

Overview

  • SUNDIALS is a SUite of Nonlinear and DIfferential/ALgebraic equation Solvers.
  • This guide will cover how to run SUNDIALS on the Cluster.
  • This is a link to the SUNDIALS Homepage.

Summary

  • SUNDIALS consists of 6 solvers that are targeted towards solving different problems
  • When running SUNDIALS, you will choose the appropriate solver, vector type (serial or parallel), and language (C or FORTRAN).
  • Then you will compile the program, specifying the appropriate libraries as flags and run it.

Walkthrough: Run SUNDIALS on the Cluster

  • This walkthrough will work on a sample program that uses the CVODE linear solver interface in the solution of a 3-species chemical kinetics problem.
  • More information about this example can be found in section 2.1 of the Example Programs for CVODE PDF provided by the Center for Applied Scientific Computing Lawrence Livermore National Laboratory.
  • For more information about CVODE, please visit the User Documentation for CVODE.
  • The cvRoberts_dns.c file can be found in the examples directory of the SUNDIALS module. To access it, run the following in your working directory:
cp /usr/local/pacerepov1/sundials/2.5.0/gcc-4.9.0/examples/cvode/serial/cvRoberts_dns.c .
  • PBS Script can be found here
  • You can transfer the files to your account on the cluster to follow along. The file transfer guide may be helpful.

Part 1: The PBS Script

#PBS -N sundialsTest
#PBS -A [Account]
#PBS -l nodes=1:ppn=2
#PBS -l pmem=2gb
#PBS -l walltime=3:00
#PBS -q inferno
#PBS -j oe
#PBS -o sundialsTest.out

cd $PBS_O_WORKDIR
module load gcc/4.9.0
module load lapack/3.6.0
module load sundials/2.5.0

gcc -lsundials_cvode -lsundials_nvecserial -lm  -llapack -lblas cvRoberts_dns.c

./a.out > result.out
  • The #PBS directives are standard, requesting 3 minutes of walltime and 1 node with 2 cores. More on #PBS directives can be found in the PBS guide
  • $PBS_O_WORKDIR is a variable that represents the directory you submit the PBS script from. Make sure the files you want to use are in the same directory you put the PBS script.
  • Output Files will also show up in this dir as well
  • module load sundials/2.5.0 loads the 2.5.0 version of SUNDIALS. To see what versions of a software are available, run module avail [Software], and load the one you want. The other modules are dependencies that must be loaded before SUNDIALS is loaded.
  • gcc -lsundials_cvode -lsundials_nvecserial -lm -llapack -lblas cvRoberts_dns.c will compile the cvRoberts_dns.c file using the specified libraries passed in with the -l flag.
  • ./a.out > result.out will print the output of running the program to result.out.

Part 2: Submit Job and Check Status

  • Make sure you're in the dir that contains the PBS Script as well as the SUNDIALS program
  • Submit as normal, with qsub <pbs script name>. In this case qsub sundials.pbs
  • Check job status with qstat -t 22182721, replacing the number with the job id returned after running qsub
  • You can delete the job with qdel 22182721 , again replacing the number with the jobid returned after running qsub

Part 3: Collecting Results

  • In the directory where you submitted the PBS script, you should see a sundialsTest.out file which contains the results of the job and results.out which contains the resulting output from compiling cvRoberts_dns.c and running ./a.out. Use cat or open the file in a text editor to take a look.
  • sundialsTest.out should just have the PBS Prologue at the top and the PBS Epilogue at the bottom with nothing in between.
  • results.out should look like this: 
3-species kinetics problem

At t = 2.6391e-01      y =  9.899653e-01    3.470564e-05    1.000000e-02
    rootsfound[] =   0   1
At t = 4.0000e-01      y =  9.851641e-01    3.386242e-05    1.480205e-02
At t = 4.0000e+00      y =  9.055097e-01    2.240338e-05    9.446793e-02
At t = 4.0000e+01      y =  7.158017e-01    9.185037e-06    2.841892e-01
At t = 4.0000e+02      y =  4.505360e-01    3.223271e-06    5.494608e-01
At t = 4.0000e+03      y =  1.832299e-01    8.944378e-07    8.167692e-01
At t = 4.0000e+04      y =  3.898902e-02    1.622006e-07    9.610108e-01
At t = 4.0000e+05      y =  4.936383e-03    1.984224e-08    9.950636e-01
At t = 4.0000e+06      y =  5.168093e-04    2.068293e-09    9.994832e-01
At t = 2.0790e+07      y =  1.000000e-04    4.000397e-10    9.999000e-01
    rootsfound[] =  -1   0
At t = 4.0000e+07      y =  5.202440e-05    2.081083e-10    9.999480e-01
At t = 4.0000e+08      y =  5.201061e-06    2.080435e-11    9.999948e-01
At t = 4.0000e+09      y =  5.258603e-07    2.103442e-12    9.999995e-01
At t = 4.0000e+10      y =  6.934511e-08    2.773804e-13    9.999999e-01

Final Statistics:
nst = 542    nfe  = 755    nsetups = 107    nfeLS = 0      nje = 11
nni = 751    ncfn = 0      netf = 22     nge = 570
  • After the result files are produced, you can move the files off the cluster, refer to the file transfer guide for help.
  • Congratulations! You successfully ran SUNDIALS on the cluster.