Updated 2020-02-04

Hive Cluster Resources

Overview

  • You can see the particular resources available on a Hive queue at any given time with pace-check-queue <insert queue name>
  • You can see even more detailed information about each node in a queue with pace-check-queue <insert queue name> -s
  • hive-all includes the nodes from all other queues.

Resource Overview of Queues

Queue Number of Nodes Number of Cores Usable Memory Per Node Total Memory Wall Time Limit
hive 400 9600 192 GB 86.2 TB 5 days
hive-gpu 16 384 384 GB 6.3 TB 3 days
hive-himem 4 116 1.5 TB 7.7 TB 5 days
hive-sas 16 384 192 GB 3.2 TB 30 days
hive-nvme 16 384 192 GB 3.2 TB 30 days
hive-nvme-sas 32 768 192 GB 6.4 TB 30 days
hive-interact 32 768 192 GB 6.2 TB 1 hour
hive-priority 256 6144 192 GB 49.0 TB 5 days
hive-all 484 11636 Contains all of above 106 TB 5 days

Detailed Node Specs

  • The following chart provides detailed specifications for the nodes that are part of the Hive Cluster:
Node Type Quantity CPU Information RAM Storage Extra Unique Specs
hive 400 Xeon 6226 CPU @ 2.70GHz 192 GB 1x512 GB SSD None
hive-gpu 16 Xeon 6226 CPU @ 2.70GHz 384 GB 1x512 GB SSD 4 x V100 NVIDIA GPU (32GB)
hive-himem 3 Xeon 6226 CPU @ 2.70GHz 1.5 TB 1x512 GB SSD None
hive-himem 1 Xeon Gold 6238M @ 2.10GHz 3.0 TB 1x512 GB SSD None
hive-sas 16 Xeon 6226 CPU @ 2.70GHz 192 GB 4x1.8 TB SAS (7.3TB Usable, RAID0) None
hive-nvme 16 Xeon 6226 CPU @ 2.70GHz 192 GB 5x1.5 TB NVMe (7.3TB Usable, RAID0) None
hive-nvme-sas 16 Xeon 6226 CPU @ 2.70GHz 192 GB 4x1.8 TB SAS (7.3TB Usable, RAID0) None
hive-nvme-sas 16 Xeon 6226 CPU @ 2.70GHz 192 GB 5x1.5 TB NVMe (7.3TB Usable, RAID0) None
hive-interact 32 Xeon 6226 CPU @ 2.70GHz 192 GB 1x512 GB SSD None
hive-priority 256 Xeon 6226 CPU @ 2.70GHz 192 GB 1x512 GB SSD None
head 2 Xeon 6226 CPU @ 2.70GHz 192 GB 2x512 GB (RAID1) None
Globus 1 Xeon 6226 CPU @ 2.70GHz 192 GB 2x512 GB (RAID1) 2x40GbE network
scheduler 1 Xeon 6226 CPU @ 2.70GHz 192 GB 2x512 GB (RAID1) & 2x2TB SSD (RAID1) None
storage 1 Xeon 6226 CPU @ 2.70GHz 192 GB 2x512 GB (RAID1) & 12x12TB SAS (ZFS) None
  • Specs common to all nodes (unless otherwise specified):
    • Operating system: Red Hat Enterprise Linux 7.6 (kernel 3.10.0-957.12.1.el7)
    • Networking Technology: 10GbE, EDR (100Gb/s) Infiniband

Detailed Specs for Other Hardware

  • The following chart provides detailed specifications for the other hardware that is a part of the Hive Cluster:
Resource Type Quantity Details
Infiniband Switch Rack 1
Mellanox CS7500 Switch 1 EDR (100Gb/s)
Line Cards 15 Contain 36 ports each
EDR ports 540 484 compute nodes directly connected
EDR ports 6 For interconnect with infastructure nodes
Mellanox QM8700 Switch 1 HDR Infiniband 200Gb/s. Infrastructure nodes and DDN system connected to this switch @ 100GB/s (EDR); 6 ports @ 100GB/s each connected to main CS7500 with 6 x 100m active optic cables.
GPFS storage rack 1
NSD Controllers 4
GS14K Embedded Storage Controller 1
SS9012 Drive Enclosure 10
SAS Drives 410 8TB 7.2K rpm (2.3PB usable)
SSD Drives 9 1.92TB
Infiniband Connections 8 EDR (100GB/s)


This material is based upon work supported by the National Science Foundation under grant number 1828187. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.