Updated 2022-01-10

Hive Testflight Environment for RHEL7.9

Testflight-Hive is designed for Hive users to test their workflows in advanced of PACE's RHEL7.9 operating system upgrade in February 2022.

Testflight-Hive Login Node

Access Testflight-Coda by ssh to login-tf-hive.pace.gatech.edu. You will arrive in your Hive home directory, with access to your Hive project & scratch storage as well.

The login node runs RHEL7.9 and has the updated PACE software stack modules, so you can use it to recompile your code in the new environment.

Modifying PBS Scripts

Researchers will need to make a few changes to the PBS scripts they generally use on Hive to use the Testflight environment. Since these are temporary changes for testing, you should copy your normal script.

There is no need to modify your module load commands, as the upgraded PACE-provided modules have the same names and versions as the existing ones. If you use the Testflight environment, you will automatically load the upgraded modules.

Choosing a Queue (-q)

Change the -q line to use a testing queue:

  • testflight for CPU jobs
  • testflight-gpu for GPU jobs

Resource Requests (-l)

  • Testflight is designed for testing code, not production, so please run only a short test to validate your software and workflow. Jobs are limited to 3 hours and 24 CPUs. Adjust your -l resource requests to be within this limit.

  • Since the primary effect of the upgrade is to MPI, if you are testing an MPI-dependent code, be sure to use multiple nodes. You can ensure multiple distinct nodes are assigned to your job by requesting more resources in total than are present on a single node (12 CPUs; 2 GPUs).

Example PBS Directives

For a simple test, swap these lines into your existing scripts instead of the ones you generally use.

To test MPI code across two full nodes for one hour, you can use these lines in place of the ones you usually use:

#PBS -q testflight
#PBS -l nodes=2:ppn=12
#PBS -l pmem=7GB
#PBS -l walltime=1:00:00

Note

Be sure to adjust any line in your code (e.g., mpirun calls) that include the number of cores available for your job, so that it matches the scheduler request.

To test a GPU+MPI job on 2 GPUs across 2 nodes for 1 hour, you can use these lines:

#PBS -q testflight-gpu
#PBS -l nodes=2:ppn=12:gpus=1
#PBS -l pmem=7GB
#PBS -l walltime=1:00:00