Updated 2021-09-21
Hive Storage and File Transfer¶
Storage¶
Important
Users have separate home, data, and scratch directories on Hive compared to other PACE systems.
-
Home
- This is your home directory, the directory you are in when you log into, and your main directory. You can create whatever files and directories you want in
~
- Is your user directory, name will be something like
/storage/home/hhive1/someuser3
but you can access it as~
- home space is backed up (includes any files or directories you make in the home directory)
- is limited by a 5gb quota (no file number limit)
- This is your home directory, the directory you are in when you log into, and your main directory. You can create whatever files and directories you want in
-
Data
~/data
is the place for any large files that need to be stored long term data sets, etc amount of storage depends on what cluster you're on and type of user. Can find amount of storage by running pace-quota (which will be covered later) and looking for the 'hard limit' line under "data" (should be upwards of 100g)
Note
~/data
does not have a user block quota, however, there is a 2 million file inode quota per user.- Each research group has a quota shared across all research members. On the Hive clusers this is 50 TB per PI, unless additional funding is specifically provided by the NSF grant funding Hive.
-
Scratch
~/scratch
is for short term data- Great for a working environment, such as moving files during a job, storing data to be used in a job that doesn't need to be on the cluster long term, or as place to store generated files from a job.
- Common workflow looks like this:
- Using a file transfer service like Globus, copy scripts and dataset into
~/scratch
folder - When job is executed, the data remains in
~/scratch
- Output and resulting generated files will show up in
~/scratch
. Then, move the important results data to your data directory (see above), or transfer them off the cluster if needed. Then empty~/scratch
and remove unneeded temporary files.
- Using a file transfer service like Globus, copy scripts and dataset into
- Each week, files older than 60 days are automatically deleted from
~/scratch
Note
~/scratch
File limit is 1 Million Files~/scratch
Storage limit is 7 TB
File Transfer¶
To move data to or from Hive, please use Globus. Globus can be used to transfer files between Hive and Phoenix or between Hive and other locations outside of PACE.
This material is based upon work supported by the National Science Foundation under grant number 1828187. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.