Updated 2023-02-14
Run Julia on the Cluster¶
Overview¶
This guide will cover how to:
- Load the Julia module
- Set up your directories to install your own Julia packages
- Setup MPI.jl to run multi-process Julia programs on the cluster
- Setup CUDA.jl to run GPU Julia programs on the cluster
Loading the Julia Module¶
To load the most recent Julia version (v1.7.2), you must first load the GCC 8.3.0 module. This can be done with either one of these:
module load gcc/8.3.0 julia/1.8.0
module load gcc julia
These two commands are equivalent, since GCC 8.3.0 and julia/1.8.0 are the default versions of the respective modules.
Note
Loading the julia/1.8.0
module will also load the openmpi/3.1.6
module, which
is compatible with MPI.jl (see below).
Setup Directories for Julia Packages¶
By default, Julia places user-installed packages in your home directory at ~/.julia
.
However, Julia packages can be quite large, and your home directory has a
relatively small storage quota (see "Storage on the
Cluster" for more info). Hence, we strongly
recommend that you create a symlink named ~/.julia
that points to your project space,
which has a much larger storage quota.
If you don't already have a ~/.julia
directory:
- Create a
.julia
directory in your project space:mkdir ~/p-<pi-username>-<number>/.julia
- Create a symlink from your project space to your home directory:
ln -s ~/p-<pi-username>-<number>/.julia ~/.julia
, e.g.,ln -s ~/p-gburdell3-0/.julia ~/.julia
(Phoenix) orln -s ~/data/.julia ~/.julia
(Hive)
If you do have an existing ~/.julia
directory:
- Move your
.julia
directory to your project space:mv ~/.julia ~/p-<pi-username>-<number>/
(Phoenix, replacingwith the username of the faculty member who provides your storage and with the volume, usually 0
) ormv ~/.julia ~/data/
(Hive) - Create a symlink from your project space to your home directory:
ln -s ~/p-<pi-username>-<number>/.julia ~/.julia
(Phoenix) orln -s ~/data/.julia ~/.julia
(Hive)
Using MPI in Julia¶
MPI.jl is an interface to the MPI message-passing library. This is a versatile way to way to write multi-process Julia programs that can scale to many nodes. The typical MPI.jl workflow requires the MPI.jl package as well as setting up the use of srun for Slurm.
PACE recommends using OpenMPI with Julia. The default MPI is MVAPICH2, so please load OpenMPI before running MPI in Julia each time.
One-Time Setup¶
Do the following on either a login or compute node, to install the MPI package and set it up to run through the Slurm scheduler, using MPIPreferences (compatible with Julia 1.8):
- Load the Julia module:
module load openmpi julia
- Start an interactive Julia session by running
julia
- Enter
using Pkg
- Enter
Pkg.add("MPIPreferences")
- Enter
using MPIPreferences
- Enter
MPIPreferences.use_system_binary(mpiexec="srun")
- Enter
Pkg.add("MPI")
- Exit Julia with
exit()
, then relaunch it if desired to run in Julia.
The SBATCH script is similar to the script for a typical MPI job. For example,
this SBATCH script will execute
06_scatterv.jl
with 4 processes.
#!/bin/bash
#SBATCH -J julia_scatterv
#SBATCH -A <your charge or tracking account here!>
#SBATCH -N 2 --ntasks-per-node=2
#SBATCH -t 10
#SBATCH -o Report-%j.out
#SBATCH --mail-type=BEGIN,END,FAIL
#SBATCH --mail-user=gburdell3@gatech.edu
cd $SLURM_SUBMIT_DIR
module load openmpi julia
export UCX_WARN_UNUSED_ENV_VARS=n
srun julia ./06-scatterv.jl
Using CUDA.jl¶
CUDA.jl is an interface for using CUDA in Julia. To install it on Phoenix, you must first login to a GPU node:
- Submit a job for an interactive compute session with
sbatch -A <your group account here!> -t 00:30:00 -N1 --ntasks-per-node=1 --mem-per-cpu=1G
. Wait for the job to to start. - Load the Julia module:
module load julia
- Start an interactive Julia session by running
julia
- Enter
]
at thejulia>
prompt to start thepkg>
prompt - Run
add CUDA
to install CUDA.jl - Optional: To quickly test if it's working, hit the backspace key to exist the
pkg>
prompt and return to thejulia>
prompt, and then runusing CUDA; CUDA.versioninfo()
After CUDA.jl
is installed, you can use it in interactive compute sessions on
GPU or submit batch jobs like usual. For example, this SBATCH script will execute peakflops.jl
on 1 GPU:
#!/bin/bash
#SBATCH -J julia_peakflops
#SBATCH -A <your group account here!>
#SBATCH -N 1 --ntasks-per-node=1
#SBATCH -t 10
#SBATCH -o Report-%j.out
#SBATCH --mail-type=BEGIN,END,FAIL
#SBATCH --mail-user=gburdell3@gatech.edu
cd $SLURM_SUBMIT_DIR
module load julia
julia ./peakflops.jl
Using Julia in Jupyter¶
To use Julia in an ondemand Jupyter notebook, follow the below steps:
- Setup the directories for Julia packages as described here.
- Setup anaconda directories as described [here] (https://docs.pace.gatech.edu/slurm-software/anaconda/#one-time-setup-phoenix-hive-and-firebird-only )
-
Create your own conda environment and install ipykernel into it
``` module load anaconda3 conda create --name
conda activate conda install ipykernel
4. Load julia on the command line ```module load julia```
5. Open Julia and install IJulia on the command line
```
julia
>>>using Pkg
>>>Pkg.add("IJulia")
>>>exit()
- Now, go to OnDemand and start a Jupyter session
- Open a new notebook with the Julia kernel 1.8.0 (Note, it's the julia kernel not the local conda env)
- You should now be able to install any additional Julia packages you need within the notebook.