Updated 2020-04-14

Storage on the Cluster

Overview

  • PACE provides a robust storage and file transfer service for users
  • This guide will go over the details of the storage system, from what directories are backed up to what the storage limits are for each directory.
  • In your home directory, there are two directories named data and scratch. Home, data, and scratch are the directories PACE provides to users.
  • You can purchase expanded storage here

Home

  • This is your home directory, the directory you are in when you log into, and your main directory. You can create whatever files and directories you want in ~
  • Is your user directory, name will be something like /nv/hp16/someuser3 but you can access it as ~

Screenshot

  • In the screeshot, you can see I am currently in my home directory, and if I list the contents using ls, you can also see the ~/data and ~/scratch directories, plus some of the directories I made myself in ~ (like my ~/scripts dir)
  • home space is backed up (includes any files or directories you make in the home directory)
  • contains ~/data and ~/scratch
  • ~ is limited by a 5gb quota (no file number limit)

Data

  • ~/data is the place for any large files that need to be stored long term
  • data sets, etc
  • amount of storage depends on what cluster you're on and type of user. Can find amount of storage by running pace-quota (which will be covered later) and looking for the 'hard limit' line under "data" (should be upwards of 100g).
  • ~/data storage limit is 2 million files/dirs for each user
  • ~/data can be expanded, purchase options and additional details can be found on the PACE participation page

Scratch

  • ~/scratch is for short term data
  • Great for a working environment, such as moving files during a job, storing data to be used in a job that doesn't need to be on the cluster long term, or as place to store generated files from a job.
  • Common workflow looks like this:
    • Using a file transfer service like Globus, copy scripts and dataset into ~/scratch folder
    • When job is executed, the data remains in ~/scratch
    • Output and resulting generated files will show up in ~/scratch. Then, move the important results data to your data directory (see above), or transfer them off the cluster if needed. Then empty ~/scratch and remove unneeded temporary files.
  • Each week, files older than 60 days are automatically deleted from ~/scratch
  • ~/scratch File limit is 1 Million Files
  • ~/scratch Storage limit is 7 TB

Check your Storage Status

  • Use pace-quota to check how much of each storage quota you have used, and how many files you have

Screenshot

  • pace-quota also gives storage limit for each directory (helpful for knowing how much space you have in ~/data)