_{Updated 2020-01-14}

How to Choose Job Resources¶

If you are using mpi, you're welcome to divide your runs into whatever nodes and ppn that you'd like, provided the product of the two is equal to total number of cores you'd like to use.
Keep in mind, not all of our nodes have the high speed networking that mpi gets its strength from.

First and foremost, it depends on your job.
An understanding of how your program handles memory is important - and finding optimizations/reducing excess memory usage can not only improve performance, but reduce your resource needs, which gets you through queue waiting times faster.
Additionally, the ways your program stores data is important - double format uses twice as much as single format, etc
Keep in mind there's usually a sacrifice in precision when changing formats, and this may or may not matter depending on application.
That said, mem is the memory for a 1 node job, pmem is for jobs where nodes > 1, where pmem is a per-core (ppn) assignment.
Total memory for the job is ppn*pmem. For a single node job, mem=ppn*pmem.
If the memory requirements for your job are high (>160GB) you're likely better off splitting it across multiple nodes.

Number of nodes is largely determined by your memory requirement
If you need (approximately >=160 GB), then your job could spend a lot of time in queues (depending on current usage) waiting for a single node with that much memory to become available.
Mpi allows you to distribute your memory requirements for a single job across multiple nodes, getting around the time spent waiting in the queue.

PBS -q is the queue.
Compute nodes belong to queues, so this determines what computers (and who else is using them) your stuff runs on.

PBS -j determines if output and error files are output, PBS -o is the filename.
It is recommended to specify both.

In general, it's a good idea to specify modules in the job script manually to make it easier to troubleshoot.
.pacemodules will automatically start this for every job and login terminal that you start, but you may decide to change this later on, which would affect all of your job scripts
Running a module purge and then your module load commands for each pbs script is also a good way to mark which versions of software you used, which is useful for when others try to reproduce your work.

This information and much more is covered in our biweekly orientation
The orientation is interactive and provides a means for you to talk to us directly about applications and problems, which is mutually beneficial
Please visit this link for more information about orientations.