Updated 2022-11-13
Link Gateway Account to Hive Cluster¶
Note
The Hive Gateway has been in production since October 2020.
Overview¶
- For users who have access to the general Hive cluster, it is possible to take advantage of additional resources with the gateway. In order to this, the user needs to create a Resource Profile.
- This guide will show you how to link your gateway account to Hive cluster as your personal resource so you can submit jobs as your PACE user and access your PACE Hive storage
Step 1: Set up a Resource Profile¶
- Go to the
Settings
tab (appears on a dropdown after clickingWorkspace
) - Click the
Group Resource Profile
icon that looks like three stacked servers in the left navbar - Click the blue
New Group Resource Profile +
button in the top right
- In the
Name
field, enter something like "[your_username]'s Resource Profile" - In the
Default SSH Credential
field, you can either use theDefault Key
that appears after clicking on the field, or you can create a new SSH Credential by clicking the plus sign next to the field
- Give the new key a name, and click
Create
.
Important
Copy the SSH key to your clipboard by selecting the copy icon next to the plus icon. You'll need this later on
- You can manage who you share this Resource Profile with by clicking the
Share
button below
- Your screen should now look something like this
- Click on the
New Compute Preference +
button and selectHive
as your compute resource
- You'll be taken to a new screen where you define the compute preferences
- In the
Login Username
field, enter your Hive cluster login username - In the
SSH Credential
field, select the same SSH Credential you selected before for the resource profile - Change the
Allocation Project Number
field to your tracking account, in the formathive-<gt-username>
. - In the
Scratch Location
field, enter your scratch location. You can find this by running this by running this commandcd ~/scratch; pwd -P
after connecting to the Hive Cluster viassh
. The path will look something like/storage/hive/scratch/N/username
whereN
is a number. - In the
Policy
section, unselect theaccess-hive
andaccess-gpu
queues. - if you are a GT user, you will get errors trying to submit to those. - Click
Save
again once taken back to the resource profile page
Step 2: Add SSH Key to authorized_keys¶
- Log in to your Hive cluster account through a terminal window
- When you are logged in, navigate to the
.ssh
folder withcd .ssh
- Open the
authorized_keys
file with a text editor of choice (you can use nano if you prefer to work in the terminal by typingnano authorized_keys
) - Paste the SSH Key you copied earlier on the last line of this file, save the file, and exit
Step 3: Submit a Job Through Your Personal Resource¶
- Go back to your workspace and select an application that you would like to submit a job for using your personal resource on Hive
- Set up the job like normal, but make sure to change the
Allocation
to the Resource Profile you set up in Step 1
- Save and Launch the job
- When the job is complete, you can access the files from your
~/scratch
directory on the Hive Cluster in addition to viewing them in the Experiment Summary page - Note: The name of the job is automatically generated, so the folder found in
~/scratch
that contains your results might have a name that is not very readable (it will most likely appear as PROCESS followed by a list of numbers)- You can use the
mv
command to rename it:mv <old_folder_name> <new_folder_name>
- You can use the