Updated 2021-01-11

Preserving Scratch Data for the Phoenix Migration

PACE will migrate your home and project (data) storage in Rich to Phoenix, and your files will be available to you there. Any files you have stored in your scratch directory in Rich (~/scratch) will not be migrated to Phoenix, and you will permanently lose access to any files there as soon as your migration begins. If you will need any data in your scratch directory after the migration, you must relocate it prior to the migration to your project storage ("data" link on your home directory). Please move only what you will need in the future.

To move these files effectively, please use rsync instead of cp, which will be much slower and cannot recover from interruptions.


Since copying a large quantity of data may take a significant amount of time, please begin at least 1 week before your migration date. While rsync is restartable, it is preferable to avoid disruption through a scheduled job.

Example job

You can submit a PBS script like this example. Please specify a queue to which you have access and set sufficient walltime. Adjust the file paths as indicated to select the data to transfer.

#PBS -N rsync-scratch
#PBS -l nodes=1:ppn=1
#PBS -l walltime=1:00:00
#PBS -q inferno
#PBS -j oe


# Make a shortcut name to the destination directory.
# Please ensure that it is within your project storage.
# In most cases, that is accessed via the "data" symlink in your home directory, as below.

# create destination directory in ~/data for scratch data to be preserved
# if it does not exist already.
[[ -d "${DEST}" ]] || mkdir -pv "${DEST}"

# For each directory in ~/scratch that you want to preserve ...
# replace "dir1", "dir2", etc. with the directories or files you wish to transfer in scratch.
# Add additional lines as needed.
rsync -avuHWP ~/scratch/dir1 "${DEST}"/
rsync -avuHWP ~/scratch/dir2 "${DEST}"/

# If the session terminates, you can restart and it will only synchronize
# what hasn't already been done.

Screen session alternative

You may choose to use a screen session instead of a job.

If you use a screen session, you may use any login node (login-s, login-d, login7-d, etc.) except novazohar (use novazohar-post1 instead) or testflight-login.

If you use a screen session, please use the following command:

rsync -avuPHW Source/ Destination/

where Source is the source file or directory to move and Destination is the location in which you want it placed.

For example, you can transfer the directory my_research to project storage to make it available after migration by running rsync -avuPHW ~/scratch/my_temporary_research ~/data/my_permanent_research.

-H preserve hardlinks that may be present in your files.

-W copies files whole.

-P shows progress while the command runs and keeps partially-transferred files.

-u skips files if a newer copy exists at the destination.

-v may be used as well, if you want detailed information on migrated files.

For additional information, you can read the rsync manual by running man rsync on PACE.