Updated 2022-03-10

Run Julia on the Cluster

Overview

This guide will cover how to:

  • Load the Julia module
  • Set up your directories to install your own Julia packages
  • Setup MPI.jl to run multi-process Julia programs on the cluster
  • Setup CUDA.jl to run GPU Julia programs on the cluster

Loading the Julia Module

To load the most recent Julia version (v1.7.2), you must first load the GCC 8.3.0 module. This can be done with either one of these:

  • module load gcc/8.3.0 julia/1.7.2
  • module load gcc julia

These two commands are equivalent, since GCC 8.3.0 and Julia 1.7.2 are the default versions of the respective modules.

Note

Loading the julia/1.7.2 module will also load the openmpi/3.1.6 module, which is compatible with MPI.jl (see below).

Setup Directories for Julia Packages

By default, Julia places user-installed packages in your home directory at ~/.julia. However, Julia packages can be quite large, and your home directory has a relatively small storage quota (see "Storage on the Cluster" for more info). Hence, we strongly recommend that you create a symlink named ~/.julia that points to your project space, which has a much larger storage quota.

If you don't already have a ~/.julia directory:

  1. Create a .julia directory in your project space: mkdir ~/p-<pi-username>-<number>/.julia
  2. Create a symlink from your project space to your home directory: ln -s ~/p-<pi-username>-<number>/.julia ~/.julia, e.g., ln -s ~/p-gburdell3-0/.julia ~/.julia (Phoenix) or ln -s ~/data/.julia ~/.julia (Hive)

If you do have an existing ~/.julia directory:

  1. Move your .julia directory to your project space: mv ~/.julia ~/p-<pi-username>-<number>/ (Phoenix, replacing with the username of the faculty member who provides your storage and with the volume, usually 0) or mv ~/.julia ~/data/ (Hive)
  2. Create a symlink from your project space to your home directory: ln -s ~/p-<pi-username>-<number>/.julia ~/.julia (Phoenix) or ln -s ~/data/.julia ~/.julia (Hive)

Using MPI.jl

MPI.jl is an iterface to the MPI message-passing library. This is a versatile way to way to write multi-process Julia programs that can scale to many nodes. The typical MPI.jl workflow requires two installations: * The MPI.jl package itself * The mpiexecjl wrapper for running MPI Julia programs

On Phoenix, do the following on either a login or compute node:

  1. Load the Julia module: module load gcc julia
  2. Start an interactive Julia session by running julia
  3. Enter ] at the julia> prompt to start the pkg> prompt
  4. Run add MPI to install MPI.jl
  5. Hit the backspace key to exist the pkg> prompt and return to the julia> prompt.
  6. Enter using MPI; MPI.install_mpiexecjl() to install the mpiexcjl wrapper.

Note

Loading Julia 1.7.2 will also load the OpenMPI 3.1.6 module, which is compatible with MPI.jl. It will also set the environment variable JULIA_MPI_BINARY="system", which instructs MPI.jl to use the system-installed MPI libraries. This configuration should work reliably on PACE. However, if you would like to install MPI.jl with different MPI libraries, see the "Configuration" section in the MPI.jl docs.

The PBS script is similar to the script for a typical MPI job. For example, this PBS script will execute 06_scatterv.jl with 4 processes. Note that we use mpiexecjl instead of mpiexec or mpirun. 

#PBS -N julia_scatterv
#PBS -A <your group account here!>
#PBS -l nodes=2:ppn=2
#PBS -l walltime=00:10:00
#PBS -j oe

cd $PBS_O_WORKDIR
module load gcc julia

mpiexecjl -np 4 julia ./06-scatterv.jl

Using CUDA.jl

CUDA.jl is an interface for using CUDA in Julia. To install it on Phoenix, you must first login to a GPU node:

  1. Submit a job for an interactive compute session with qsub -I -l nodes=1:ppn=1:gpus=1,walltime=00:30:00 -A <your group account here!>. Wait for the job to to start.
  2. Load the Julia module: module load gcc julia
  3. Start an interactive Julia session by running julia
  4. Enter ] at the julia> prompt to start the pkg> prompt
  5. Run add CUDA to install CUDA.jl
  6. Optional: To quickly test if it's working, hit the backspace key to exist the pkg> prompt and return to the julia> prompt, and then run using CUDA; CUDA.versioninfo()

After CUDA.jl is installed, you can use it in interactive compute sessions on GPU or submit batch jobs like usual. For example, this PBS script will execute peakflops.jl on 1 GPU:

#PBS -N julia_peakflops
#PBS -A <your group account here!>
#PBS -l nodes=1:ppn=1:gpus=1
#PBS -l walltime=00:10:00
#PBS -j oe

cd $PBS_O_WORKDIR
module load gcc julia

julia ./peakflops.jl