Updated 2022-01-14

Phoenix Storage Structure

Overview

Phoenix Storage

  • Use the pace-quota command on Phoenix to check your storage utilization.
  • Storage is split into 3 main types (directories):
    • home and scratch
    • The naming scheme for project storage will be p-<pi-username>-<number>, so for example p-jdoe4-0

Tip

The Home directory quota is 10GB. It is very important to limit usage of the home directory since there are many applications that must be able to write files in the home directory to function.

Home

  • This is your home directory, the directory you are in when you log in, and your main directory. You can create whatever files and directories you want in ~
  • Is your user directory, name will be something like /storage/home/hcoda1/0/someuser3 but you can access it as ~
  • home space is backed up (includes any files or directories you make in the home directory)
  • Contains symbolic links to your project storage directory and ~/scratch
  • ~ is limited by a 10GB quota. Home directories are also limited to 1 million files or directories.

Project Storage

  • The naming scheme for project storage is p-<pi-username>-<number>, so for example p-jdoe4-0, in most cases.
    • The p represents "project", jdoe4 is the responsible PI's username, and the number 0 represents how many projects are associated with your PI.
    • The PI username in the name of the project indicates the PI (Georgia Tech faculty member) who is responsible for the storage project and for any data contained in it. When a user leaves Georgia Tech, this PI will receive the data contained in the user's project storage.
    • Some project storage locations start with d followed by the abbreviation for a school, college, or unit, e.g., d-chem-0. In this case, the unit, rather than a specific faculty member, is responsible for the data.
  • A user may have access to multiple project storage locations, especially if that user has multiple supervisors. The PI username in each one indicates which PI is responsible for the data in that directory.
  • The symlink for your project storage with the structure p-<pi-username>-<number> points to /storage/coda1/p-<pi-username>/<number>/<username>
  • Project Storage is the place for any large files that need to be stored long term data sets, etc
  • The quota for project storage is shared by all members of a research group (not set by user) and is determined by the PI's purchased amount.
  • Each research group has a shared directory in their project storage, which is accessible to anyone in the group. This directory may be helpful holding shared datasets or code. It is located at /storage/coda1/p-<pi-username>/<number>/shared, following the same pattern as your personal project storage directory, but replacing the last segment with shared instead. If you would like a convenient symbolic link to your shared directory in your home directory, please visit our Symlink Guide for assistance.
  • Project Storage: does not have a limit on number of files/directories for each user or research group
  • Project Storage can be expanded. Each GT academic or research faculty member receives 1 TB funded by the Institute for their storage project. If a research group needs more storage than that provided by PACE, the PI may purchase additional space. Purchase options and additional details can be found on the PACE participation page. All storage billing on Phoenix is in terms of TB*months of credit. For GT faculty, storage costs $6.67/TB/month, and faculty can choose lump sum (buy credits in advance) or monthly (be billed at the start of each month for their quota from the previous month to the worktag of their choosing) storage.
  • If a PI exhausts storage funds and lacks sufficient credit to cover their quota for a full month (and has not set up a monthly payment), then their project quota will be set to 1 TB on the 1st of the month, even if credits remain for a partial month. Note that any jobs that crash at the time of quota setting due to insufficient quota will not be eligible for a refund.
  • Should needs change, PIs who choose the lump sum payment option may adjust the rate at which their prepaid balance declines by raising their quota and moving the expiration date sooner or lowering the quota and moving the expiration date farther into the future.

Scratch

  • ~/scratch is for short term data and is not backed up, so important data should not be stored there.
  • Great for a working environment, such as moving files during a job, storing data to be used in a job that doesn't need to be on the cluster long term, or as place to store generated files from a job.
  • Common workflow looks like this:
    • Using a file transfer service like Globus, copy scripts and dataset into ~/scratch folder
    • When job is executed, the data remains in ~/scratch
    • Output and resulting generated files will show up in ~/scratch. Then, move the important results data to your project directory (see above), or transfer them off the cluster if needed. Then empty ~/scratch and remove unneeded temporary files.
  • Each week, files older than 60 days are automatically deleted from ~/scratch
  • ~/scratch Storage limit is 15 TB
  • ~/scratch: There is a limit of 1 million files or directories.

File Transfer and Sharing

  • To move data to or from Phoenix, please use Globus. Globus can be used to transfer files between Hive and Phoenix or between Phoenix and other locations outside of PACE.
  • To share data between lab members, copying (cp) or moving (mv) it to the Shared Project Storage directory will give everyone in the group access.
    • For example: cp -r /storage/coda1/home/gburdell3/directory-to-share /storage/coda1/p-<pi-username>/<number>/shared/ will give all users in the group access to the directory-to-share.
    • It may also be necessary to make the files writable by the group, via chmod u+w /storage/coda1/p-<pi-username>/<number>/shared/<filename>