Updated 2021-01-20

Storage on the Cluster

Overview

  • PACE provides a robust storage and file transfer service for users
  • There are different storage setups for Hive and Phoenix. Below, there are links to find out more about the file system for each.

Phoenix Storage

  • Phoenix has home/, project/, and scratch/ directories
  • More details and storage limits for these directories can be found in the Phoenix Storage Guide

Hive Storage

  • Hive has home/, data, and scratch/ directories
  • Home

    • This is your home directory, the directory you are in when you log into, and your main directory. You can create whatever files and directories you want in ~
    • Is your user directory, name will be something like /storage/home/hhive1/someuser3 but you can access it as ~
    • home space is backed up (includes any files or directories you make in the home directory)
    • is limited by a 5gb quota (no file number limit)
  • Data

    • ~/data is the place for any large files that need to be stored long term data sets, etc amount of storage depends on what cluster you're on and type of user. Can find amount of storage by running pace-quota (which will be covered later) and looking for the 'hard limit' line under "data" (should be upwards of 100g)

    Note

    • ~/data storage limit is 2 million files/dirs for each user
  • Scratch

    • ~/scratch is for short term data
    • Great for a working environment, such as moving files during a job, storing data to be used in a job that doesn't need to be on the cluster long term, or as place to store generated files from a job.
    • Common workflow looks like this:
      • Using a file transfer service like Globus, copy scripts and dataset into ~/scratch folder
      • When job is executed, the data remains in ~/scratch
      • Output and resulting generated files will show up in ~/scratch. Then, move the important results data to your data directory (see above), or transfer them off the cluster if needed. Then empty ~/scratch and remove unneeded temporary files.
    • Each week, files older than 60 days are automatically deleted from ~/scratch

    Note

    • ~/scratch File limit is 1 Million Files
    • ~/scratch Storage limit is 7 TB

Check your Storage Status

  • Use pace-quota to check how much of each storage quota you have used, and how many files you have
  • pace-quota also gives storage limit for each directory (helpful for knowing how much space you have in ~/data)