Updated 2023-05-11
Hive Cluster Resources¶
Overview¶
- You can see the particular resources available on a Hive partition at any given time with
pace-check-queue <insert partition name> - You can see even more detailed information about each node in a partition with
pace-check-queue <insert queue name> -s
Resource Overview of Partitions¶
Most of the 484 Hive nodes have 24 cores.
| Partition | Number of Nodes | Usable Memory Per Node | Wall Time Limit |
|---|---|---|---|
| hive | 428 | 192 GB | 5 days |
| hive-gpu | 16 | 384 GB | 3 days |
| hive-gpu-short | 16 | 384 GB | 12 hours |
| hive-himem | 4 | 1.5 TB (3) 3.0 TB (1) | 5 days |
| hive-sas | 16 | 192 GB | 30 days |
| hive-nvme | 16 | 192 GB | 30 days |
| hive-nvme-sas | 32 | 192 GB | 30 days |
| hive-interact | 432 | 192 GB | 1 hour |
Detailed Node Specs¶
- The following chart provides detailed specifications for the nodes that are part of the Hive Cluster:
| Node Type | Quantity | CPU Information | RAM | Storage | Extra Unique Specs |
|---|---|---|---|---|---|
| hive | 428 | Xeon 6226 CPU @ 2.70GHz | 192 GB | 1x512 GB SSD | None |
| hive-gpu | 16 | Xeon 6226 CPU @ 2.70GHz | 384 GB | 1x512 GB SSD | 4 x V100 NVIDIA GPU (16GB) |
| hive-gpu-short | 16 | Xeon 6226 CPU @ 2.70GHz | 384 GB | 1x512 GB SSD | 4 x V100 NVIDIA GPU (16GB) |
| hive-himem | 3 | Xeon 6226 CPU @ 2.70GHz | 1.5 TB | 1x512 GB SSD | None |
| hive-himem | 1 | Xeon Gold 6238M @ 2.10GHz | 3.0 TB | 1x512 GB SSD | None |
| hive-sas | 16 | Xeon 6226 CPU @ 2.70GHz | 192 GB | 4x1.8 TB SAS (7.3TB Usable, RAID0) | None |
| hive-nvme | 16 | Xeon 6226 CPU @ 2.70GHz | 192 GB | 5x1.5 TB NVMe (7.3TB Usable, RAID0) | None |
| hive-nvme-sas | 32 | Xeon 6226 CPU @ 2.70GHz | 192 GB | 4x1.8 TB SAS (7.3TB Usable, RAID0) (16) OR 5x1.5 TB NVMe (7.3TB Usable, RAID0) (16) | None |
| hive-interact | 432 | Xeon 6226 CPU @ 2.70GHz | 192 GB | 1x512 GB SSD | None |
| head | 2 | Xeon 6226 CPU @ 2.70GHz | 192 GB | 2x512 GB (RAID1) | None |
| Globus | 1 | Xeon 6226 CPU @ 2.70GHz | 192 GB | 2x512 GB (RAID1) | 2x40GbE network |
| scheduler | 1 | Xeon 6226 CPU @ 2.70GHz | 192 GB | 2x512 GB (RAID1) & 2x2TB SSD (RAID1) | None |
| storage | 1 | Xeon 6226 CPU @ 2.70GHz | 192 GB | 2x512 GB (RAID1) & 12x12TB SAS (ZFS) | None |
- Specs common to all nodes (unless otherwise specified):
- Operating system: Red Hat Enterprise Linux
- Networking Technology: 10GbE, EDR (100Gb/s) Infiniband
Detailed Specs for Other Hardware¶
- The following chart provides detailed specifications for the other hardware that is a part of the Hive Cluster:
| Resource Type | Quantity | Details |
|---|---|---|
| Infiniband Switch Rack | 1 | |
| Mellanox CS7500 Switch | 1 | EDR (100Gb/s) |
| Line Cards | 15 | Contain 36 ports each |
| EDR ports | 540 | 484 compute nodes directly connected |
| EDR ports | 6 | For interconnect with infastructure nodes |
| Mellanox QM8700 Switch | 1 | HDR Infiniband 200Gb/s. Infrastructure nodes and DDN system connected to this switch @ 100GB/s (EDR); 6 ports @ 100GB/s each connected to main CS7500 with 6 x 100m active optic cables. |
| GPFS storage rack | 1 | |
| NSD Controllers | 4 | |
| GS14K Embedded Storage Controller | 1 | |
| SS9012 Drive Enclosure | 10 | |
| SAS Drives | 410 | 8TB 7.2K rpm (2.3PB usable) |
| SSD Drives | 9 | 1.92TB |
| Infiniband Connections | 8 | EDR (100GB/s) |
This material is based upon work supported by the National Science Foundation under grant number 1828187. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.