Updated 2023-02-14

Run Julia on the Cluster

Overview

This guide will cover how to:

  • Load the Julia module
  • Set up your directories to install your own Julia packages
  • Setup MPI.jl to run multi-process Julia programs on the cluster
  • Setup CUDA.jl to run GPU Julia programs on the cluster

Loading the Julia Module

To load the most recent Julia version (v1.7.2), you must first load the GCC 8.3.0 module. This can be done with either one of these:

  • module load gcc/8.3.0 julia/1.8.0
  • module load gcc julia

These two commands are equivalent, since GCC 8.3.0 and julia/1.8.0 are the default versions of the respective modules.

Note

Loading the julia/1.8.0 module will also load the openmpi/3.1.6 module, which is compatible with MPI.jl (see below).

Setup Directories for Julia Packages

By default, Julia places user-installed packages in your home directory at ~/.julia. However, Julia packages can be quite large, and your home directory has a relatively small storage quota (see "Storage on the Cluster" for more info). Hence, we strongly recommend that you create a symlink named ~/.julia that points to your project space, which has a much larger storage quota.

If you don't already have a ~/.julia directory:

  1. Create a .julia directory in your project space: mkdir ~/p-<pi-username>-<number>/.julia
  2. Create a symlink from your project space to your home directory: ln -s ~/p-<pi-username>-<number>/.julia ~/.julia, e.g., ln -s ~/p-gburdell3-0/.julia ~/.julia (Phoenix) or ln -s ~/data/.julia ~/.julia (Hive)

If you do have an existing ~/.julia directory:

  1. Move your .julia directory to your project space: mv ~/.julia ~/p-<pi-username>-<number>/ (Phoenix, replacing with the username of the faculty member who provides your storage and with the volume, usually 0) or mv ~/.julia ~/data/ (Hive)
  2. Create a symlink from your project space to your home directory: ln -s ~/p-<pi-username>-<number>/.julia ~/.julia (Phoenix) or ln -s ~/data/.julia ~/.julia (Hive)

Using MPI in Julia

MPI.jl is an interface to the MPI message-passing library. This is a versatile way to way to write multi-process Julia programs that can scale to many nodes. The typical MPI.jl workflow requires the MPI.jl package as well as setting up the use of srun for Slurm.

PACE recommends using OpenMPI with Julia. The default MPI is MVAPICH2, so please load OpenMPI before running MPI in Julia each time.

One-Time Setup

Do the following on either a login or compute node, to install the MPI package and set it up to run through the Slurm scheduler, using MPIPreferences (compatible with Julia 1.8):

  1. Load the Julia module: module load openmpi julia
  2. Start an interactive Julia session by running julia
  3. Enter using Pkg
  4. Enter Pkg.add("MPIPreferences")
  5. Enter using MPIPreferences
  6. Enter MPIPreferences.use_system_binary(mpiexec="srun")
  7. Enter Pkg.add("MPI")
  8. Exit Julia with exit(), then relaunch it if desired to run in Julia.

The SBATCH script is similar to the script for a typical MPI job. For example, this SBATCH script will execute 06_scatterv.jl with 4 processes. 

#!/bin/bash
#SBATCH -J julia_scatterv
#SBATCH -A <your charge or tracking account here!>
#SBATCH -N 2 --ntasks-per-node=2
#SBATCH -t 10
#SBATCH -o Report-%j.out
#SBATCH --mail-type=BEGIN,END,FAIL
#SBATCH --mail-user=gburdell3@gatech.edu

cd $SLURM_SUBMIT_DIR
module load openmpi julia

export UCX_WARN_UNUSED_ENV_VARS=n

srun julia ./06-scatterv.jl

Using CUDA.jl

CUDA.jl is an interface for using CUDA in Julia. To install it on Phoenix, you must first login to a GPU node:

  1. Submit a job for an interactive compute session with sbatch -A <your group account here!> -t 00:30:00 -N1 --ntasks-per-node=1 --mem-per-cpu=1G. Wait for the job to to start.
  2. Load the Julia module: module load julia
  3. Start an interactive Julia session by running julia
  4. Enter ] at the julia> prompt to start the pkg> prompt
  5. Run add CUDA to install CUDA.jl
  6. Optional: To quickly test if it's working, hit the backspace key to exist the pkg> prompt and return to the julia> prompt, and then run using CUDA; CUDA.versioninfo()

After CUDA.jl is installed, you can use it in interactive compute sessions on GPU or submit batch jobs like usual. For example, this SBATCH script will execute peakflops.jl on 1 GPU:

#!/bin/bash
#SBATCH -J julia_peakflops
#SBATCH -A <your group account here!>
#SBATCH -N 1 --ntasks-per-node=1
#SBATCH -t 10
#SBATCH -o Report-%j.out
#SBATCH --mail-type=BEGIN,END,FAIL
#SBATCH --mail-user=gburdell3@gatech.edu

cd $SLURM_SUBMIT_DIR
module load julia

julia ./peakflops.jl

Using Julia in Jupyter

To use Julia in an ondemand Jupyter notebook, follow the below steps:

  1. Setup the directories for Julia packages as described here.
  2. Setup anaconda directories as described [here] (https://docs.pace.gatech.edu/slurm-software/anaconda/#one-time-setup-phoenix-hive-and-firebird-only )

  3. Create your own conda environment and install ipykernel into it

    ``` module load anaconda3 conda create --name conda activate conda install ipykernel


4. Load julia on the command line ```module load julia```

5. Open Julia and install IJulia on the command line

    ```
julia
>>>using Pkg
>>>Pkg.add("IJulia")
>>>exit()
  1. Now, go to OnDemand and start a Jupyter session
  2. Open a new notebook with the Julia kernel 1.8.0 (Note, it's the julia kernel not the local conda env)
  3. You should now be able to install any additional Julia packages you need within the notebook.