Updated 2020-12-30

Use SCP on Linux/Mac to Transfer Files

Overview

  • You can send files to whatever dir you want on the cluster, (home / personal dirs in home, scratch, or data) altough some directories might be better than others for certain files.
  • Ex: if you want to upload a very large data set that you only want to use for a couple computations, upload it to /scratch
  • This guide uses the secure tool scp which stands for Secure Copy. Its easy to use, safe, and reliable, and is run in the terminal. However, it will be noticeably slower than globus if transfering large files

From Local Computer to Cluster (Onto Cluster)

  • Say I wanted to send a PBS script, a python file, and data set in the form of a .csv file to the cluster.
  • Im going to send them to scratch, since I plan on running the python file on the data set only a couple times, gathering the results, and then deleting the files off scratch (I don't plan on using them anymore)
  • The first step is to make sure you are in the directory of the files you want to send
  • The template to send files from a local machine to a remote host is:
# For Phoenix
scp file.txt someuser3@login-phoenix.pace.gatech.edu:/path/to/destination

# For Hive:
scp file.txt someuser3@login-hive.pace.gatech.edu:/path/to/destination

# What I would run for the example (on Phoenix)
scp script.pbs classifier.py data.csv shollister7@login-phoenix.pace.gatech.edu:~/scratch

From Cluster to Personal Computer (Off of Cluster)

  • Lets say a job had just finished running, and had created a log file with its results.
  • To move those result files off of the cluster to your computer, you can use scp.
  • For this scenario I will be moving weather.csv , a csv file that contains the results of a python program I have run on the cluster, to my personal laptop.
  • The template to send files from the cluster to a local machine is this:
scp someuser3@login-phoenix.pace.gatech.edu:~/path_of_file_to_transfer ~/destination_to_transfer_to

#What I would run for the example (transfer the csv off the cluster to a folder on my laptop called "ml")
scp shollister7@login-phoenix.pace.gatech.edu:~/data/weather.csv ~/ml

Moving Directories

  • To move a directory, simply use the -r flag which tells scp to copy the directory and anything contained in it recursively
  • Copying /scripts dir From Remote (cluster) to /PACE-GT on local machine (laptop):
scp -r shollister7@login-phoenix.pace.gatech.edu:~/scripts ~/PACE-GT