Updated 2020-04-10

Submit Jobs to Hive Queues

Summary

  1. Navigate to your working directory
  2. Create a PBS Script or download the template found below here.
  3. Choose the appropriate queue to submit your job to and fill in the fields you are using.
  4. Submit your job with qsub job_name.pbs replacing job_name.pbs with the name of your PBS Script.

Note

For more information about how to create a PBS Script, please refer to the PBS Scripting Guide. All information found there applies to the Hive Cluster except for the queues that a job may be submitted to (a list of the Hive Cluster queues can be found towards the end of this page).

Example

  • The following is a PBS Script template that serves as an example.
  • You can download the script here. Make sure to edit the appropriate fields before attempting to submit a job.

Warning

When creating a PBS script, make sure to include a blank line at the end or else the last line in the script may not be executed.

#PBS -N [INSERT NAME]           # job name
#PBS -l nodes=2:ppn=4           # number of nodes and cores per node required
#PBS -l pmem=2gb                # memory per core
#PBS -l walltime=15:00          # duration of the job (ex: 15 min)
#PBS -q [INSERT QUEUE]          # queue name (where job is submitted)
#PBS -j oe                      # combine output and error messages into 1 file
#PBS -o [INSERT NAME].out       # output file name
#PBS -m abe                     # event notification, set to email on start, end, or fail
#PBS -M [INSERT EMAIL ADDRESS]  # email to send notifications to

cd $PBS_O_WORKDIR 

for i in {1..5}
do 
    echo "Test $i"
done

  • For this example, the name was hiveTest and the queue was hive-all.
  • After the fields are filled in, the jobs can be submitted with qsub hiveTest.pbs.
  • hiveTest.out should look like this:
---------------------------------------
Begin PBS Prologue Fri Sep  6 15:50:27 EDT 2019
Job ID:     113.sched-hive.pace.gatech.edu
User ID:    svemuri8
Job name:   hiveTest
Queue:      hive-all
End PBS Prologue Fri Sep  6 15:50:27 EDT 2019
---------------------------------------
Test 1
Test 2
Test 3
Test 4
Test 5
---------------------------------------
Begin PBS Epilogue Fri Sep  6 15:50:27 EDT 2019
Job ID:     113.sched-hive.pace.gatech.edu
User ID:    svemuri8
Job name:   hiveTest
Resources:  nodes=2:ppn=4,pmem=2gb,walltime=00:15:00,neednodes=2:ppn=4
Rsrc Used:  cput=00:00:00,vmem=0kb,walltime=00:00:00,mem=0kb,energy_used=0
Queue:      hive-all
Nodes:
atl1-1-01-010-10.pace.gatech.edu
End PBS Epilogue Fri Sep  6 15:50:27 EDT 2019
---------------------------------------

Choices for Queues

  • Please choose the appropriate queue from the ones listed below for your job.
  • To submit jobs to the chosen queue, simply place the name of the queue after #PBS -q in the PBS Script.
  • The example scripts shown below have the email notification flags omitted.

hive

  • Used for jobs that don't require any special resources
  • Example PBS Script:
#PBS -N hiveTest                # job name
#PBS -l nodes=1:ppn=1           # number of nodes and cores per node required
#PBS -l pmem=1gb                # memory per core
#PBS -l walltime=15:00          # duration of the job (ex: 15 min)
#PBS -q hive                    # queue name (where job is submitted)
#PBS -j oe                      # combine output and error messages into 1 file
#PBS -o hiveTest.out            # output file name

cd $PBS_O_WORKDIR 

for i in {1..5}
do 
    echo "Test $i"
done

hive-gpu

  • Used for jobs that make use of GPUs
  • Jobs allow for no more than 6 CPUs per GPU to prevent idle GPU resources
  • Make sure to specify the number of GPUs needed in addition to other resource requests
  • Example PBS Script:
#PBS -N hivegpuTest             # job name
#PBS -l nodes=1:ppn=1:gpus=1    # number of nodes, cores per node, and gpus required
#PBS -l pmem=1gb                # memory per core
#PBS -l walltime=15:00          # duration of the job (ex: 15 min)
#PBS -q hive-gpu                # queue name (where job is submitted)
#PBS -j oe                      # combine output and error messages into 1 file
#PBS -o hivegpuTest.out         # output file name

cd $PBS_O_WORKDIR 

for i in {1..5}
do 
    echo "Test $i"
done

hive-gpu-short

  • Used for short (<12 hours) jobs that make use of GPUs
  • Jobs allow for no more than 6 CPUs per GPU to prevent idle GPU resources
  • Make sure to specify the number of GPUs needed in addition to other resource requests
  • Example PBS Script:
#PBS -N hivegpushortTest        # job name
#PBS -l nodes=1:ppn=1:gpus=1    # number of nodes, cores per node, and gpus required
#PBS -l pmem=1gb                # memory per core
#PBS -l walltime=15:00          # duration of the job (ex: 15 min)
#PBS -q hive-gpu-short          # queue name (where job is submitted)
#PBS -j oe                      # combine output and error messages into 1 file
#PBS -o hivegpushortTest.out    # output file name

cd $PBS_O_WORKDIR

for i in {1..5}
do
    echo "Test $i"
done

hive-himem

  • Used for jobs that have high memory requirements
  • Jobs can use up to 1.5 TB of memory per node
  • Example PBS Script:
#PBS -N hivehimemTest           # job name
#PBS -l nodes=1:ppn=4           # number of nodes and cores per node required
#PBS -l pmem=1tb                # memory per core
#PBS -l walltime=45:00          # duration of the job (ex: 15 min)
#PBS -q hive-himem              # queue name (where job is submitted)
#PBS -j oe                      # combine output and error messages into 1 file
#PBS -o hivehimemTest.out       # output file name

cd $PBS_O_WORKDIR

for i in {1..5}
do
    echo "Test $i"
done

hive-sas

  • Used for jobs that need to offload file I/O off of the main GPFS system or need to transfer data into and out of storage quickly.
  • Leverages larger local storage on compute-nodes.
  • Example PBS Script:
#PBS -N hivesasTest             # job name
#PBS -l nodes=1:ppn=1           # number of nodes and cores per node required
#PBS -l pmem=1gb                # memory per core
#PBS -l walltime=15:00          # duration of the job (ex: 15 min)
#PBS -q hive-sas                # queue name (where job is submitted)
#PBS -j oe                      # combine output and error messages into 1 file
#PBS -o hivesasTest.out         # output file name

cd $PBS_O_WORKDIR
cd ${TMPDIR}
fallocate -l 1G mytest.inp      # create a 1GB file
cp mytest.inp mytest.out        # simulate creation of output data
cp mytest.out $PBS_O_WORKDIR    # copy relevant data back
exit

# The $TMPDIR wil be removed at the end of the job, so make sure to move important data back


hive-nvme

  • Uses Non Volatile Memory Express technology to allow for significant improvements in IOPS while consuming less power.
  • Leverages larger local storage on compute nodes.
  • Example PBS Script:
#PBS -N hivenvmeTest            # job name
#PBS -l nodes=1:ppn=1           # number of nodes and cores per node required
#PBS -l pmem=1gb                # memory per core
#PBS -l walltime=15:00          # duration of the job (ex: 15 min)
#PBS -q hive-nvme               # queue name (where job is submitted)
#PBS -j oe                      # combine output and error messages into 1 file
#PBS -o hivenvmeTest.out        # output file name

cd $PBS_O_WORKDIR
cd ${TMPDIR}
fallocate -l 1G mytest.inp      # create a 1GB file
cp mytest.inp mytest.out        # simulate creation of output data
cp mytest.out $PBS_O_WORKDIR    # copy relevant data back
exit

# The $TMPDIR wil be removed at the end of the job, so make sure to move important data back

hive-nvme-sas

  • A hybrid queue providing the storage benefits of both the hive-nvme and hive-sas queues
  • Leverages larger local storage on compute nodes.
  • Example PBS Script:
#PBS -N hivenvmesasTest         # job name
#PBS -l nodes=1:ppn=1           # number of nodes and cores per node required
#PBS -l pmem=1gb                # memory per core
#PBS -l walltime=15:00          # duration of the job (ex: 15 min)
#PBS -q hive-nvme-sas           # queue name (where job is submitted)
#PBS -j oe                      # combine output and error messages into 1 file
#PBS -o hivenvmeTest.out        # output file name

cd $PBS_O_WORKDIR
cd ${TMPDIR}
fallocate -l 1G mytest.inp      # create a 1GB file
cp mytest.inp mytest.out        # simulate creation of output data
cp mytest.out $PBS_O_WORKDIR    # copy relevant data back
exit

# The $TMPDIR wil be removed at the end of the job, so make sure to move important data back

hive-interact

  • A queue dedicated for rapid development, testing, and debugging of applications
  • Users will not have to compete with longer batch job submissions as on other queues
  • Example job run command
qsub -I -q hive-interact -l nodes=1:ppn=1,walltime=1:00:00

hive-priority

  • A special reservation-based queue for time-sensitive research deadlines
  • Permission is granted based on consensus of the Hive governing committee
  • Example PBS Script:
#PBS -N hivepriorityTest        # job name
#PBS -l nodes=1:ppn=4           # number of nodes and cores per node required
#PBS -l pmem=1gb                # memory per core
#PBS -l walltime=1:00:00        # duration of the job (ex: 15 min)
#PBS -q hive-priority           # queue name (where job is submitted)
#PBS -j oe                      # combine output and error messages into 1 file
#PBS -o hivepriorityTest.out    # output file name

cd $PBS_O_WORKDIR

for i in {1..5}
do
    echo "Test $i"
done

hive-all

  • Used to access all resources in the Hive Cluster
  • Example PBS Script:
#PBS -N hiveallTest             # job name
#PBS -l nodes=1:ppn=4           # number of nodes and cores per node required
#PBS -l pmem=1gb                # memory per core
#PBS -l walltime=1:00:00        # duration of the job (ex: 15 min)
#PBS -q hive-all                # queue name (where job is submitted)
#PBS -j oe                      # combine output and error messages into 1 file
#PBS -o hiveallTest.out         # output file name

cd $PBS_O_WORKDIR 

for i in {1..5}
do 
    echo "Test $i"
done


This material is based upon work supported by the National Science Foundation under grant number 1828187. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.