Updated 2022-03-10
Run Julia on the Cluster¶
Overview¶
This guide will cover how to:
- Load the Julia module
- Set up your directories to install your own Julia packages
- Setup MPI.jl to run multi-process Julia programs on the cluster
- Setup CUDA.jl to run GPU Julia programs on the cluster
Loading the Julia Module¶
To load the most recent Julia version (v1.7.2), you must first load the GCC 8.3.0 module. This can be done with either one of these:
module load gcc/8.3.0 julia/1.7.2
module load gcc julia
These two commands are equivalent, since GCC 8.3.0 and Julia 1.7.2 are the default versions of the respective modules.
Note
Loading the julia/1.7.2
module will also load the openmpi/3.1.6
module, which
is compatible with MPI.jl (see below).
Setup Directories for Julia Packages¶
By default, Julia places user-installed packages in your home directory at ~/.julia
.
However, Julia packages can be quite large, and your home directory has a
relatively small storage quota (see "Storage on the
Cluster" for more info). Hence, we strongly
recommend that you create a symlink named ~/.julia
that points to your project space,
which has a much larger storage quota.
If you don't already have a ~/.julia
directory:
- Create a
.julia
directory in your project space:mkdir ~/p-<pi-username>-<number>/.julia
- Create a symlink from your project space to your home directory:
ln -s ~/p-<pi-username>-<number>/.julia ~/.julia
, e.g.,ln -s ~/p-gburdell3-0/.julia ~/.julia
(Phoenix) orln -s ~/data/.julia ~/.julia
(Hive)
If you do have an existing ~/.julia
directory:
- Move your
.julia
directory to your project space:mv ~/.julia ~/p-<pi-username>-<number>/
(Phoenix, replacingwith the username of the faculty member who provides your storage and with the volume, usually 0
) ormv ~/.julia ~/data/
(Hive) - Create a symlink from your project space to your home directory:
ln -s ~/p-<pi-username>-<number>/.julia ~/.julia
(Phoenix) orln -s ~/data/.julia ~/.julia
(Hive)
Using MPI.jl¶
MPI.jl is an iterface to the
MPI message-passing library. This is a versatile way to way to write multi-process
Julia programs that can scale to many nodes. The typical MPI.jl workflow
requires two installations:
* The MPI.jl package itself
* The mpiexecjl
wrapper for running MPI Julia programs
On Phoenix, do the following on either a login or compute node:
- Load the Julia module:
module load gcc julia
- Start an interactive Julia session by running
julia
- Enter
]
at thejulia>
prompt to start thepkg>
prompt - Run
add MPI
to install MPI.jl - Hit the backspace key to exist the
pkg>
prompt and return to thejulia>
prompt. - Enter
using MPI; MPI.install_mpiexecjl()
to install thempiexcjl
wrapper.
Note
Loading Julia 1.7.2 will also load the OpenMPI 3.1.6 module, which
is compatible with MPI.jl. It will also set the environment variable JULIA_MPI_BINARY="system"
, which instructs MPI.jl to use the system-installed
MPI libraries. This configuration should work reliably on PACE. However, if you would like to install MPI.jl with different MPI libraries, see the
"Configuration" section in the MPI.jl docs.
The PBS script is similar to the script for a typical MPI job. For example,
this PBS script will execute
06_scatterv.jl
with 4 processes. Note that we use mpiexecjl
instead of mpiexec
or mpirun
.
#PBS -N julia_scatterv
#PBS -A <your group account here!>
#PBS -l nodes=2:ppn=2
#PBS -l walltime=00:10:00
#PBS -j oe
cd $PBS_O_WORKDIR
module load gcc julia
mpiexecjl -np 4 julia ./06-scatterv.jl
Using CUDA.jl¶
CUDA.jl is an interface for using CUDA in Julia. To install it on Phoenix, you must first login to a GPU node:
- Submit a job for an interactive compute session with
qsub -I -l nodes=1:ppn=1:gpus=1,walltime=00:30:00 -A <your group account here!>
. Wait for the job to to start. - Load the Julia module:
module load gcc julia
- Start an interactive Julia session by running
julia
- Enter
]
at thejulia>
prompt to start thepkg>
prompt - Run
add CUDA
to install CUDA.jl - Optional: To quickly test if it's working, hit the backspace key to exist the
pkg>
prompt and return to thejulia>
prompt, and then runusing CUDA; CUDA.versioninfo()
After CUDA.jl
is installed, you can use it in interactive compute sessions on
GPU or submit batch jobs like usual. For example, this PBS script will execute peakflops.jl
on 1 GPU:
#PBS -N julia_peakflops
#PBS -A <your group account here!>
#PBS -l nodes=1:ppn=1:gpus=1
#PBS -l walltime=00:10:00
#PBS -j oe
cd $PBS_O_WORKDIR
module load gcc julia
julia ./peakflops.jl