Use SCP on Linux/Mac to Transfer Files¶
- Every user has access to the datamover node which allows for fast file transfer, and prevents slow down from sending files directly to the head node.
- You can send files to whatever dir you want on the cluster, (home / personal dirs in home, scratch, or data) altough some directories might be better than others for certain files.
- Ex: if you want to upload a very large data set that you only want to use for a couple computations, upload it to
- This guide uses the secure tool
scpwhich stands for Secure Copy. Its easy to use, safe, and reliable, and is run in the terminal. However, it will be noticeably slower than globus if transfering large files
From Local Computer to Cluster (Onto Cluster)¶
- Say I wanted to send a PBS script, a python file, and data set in the form of a .csv file to the cluster.
- Im going to send them to scratch, since I plan on running the python file on the data set only a couple times, gathering the results, and then deleting the files off scratch (I don't plan on using them anymore)
- The first step is to make sure you are in the directory of the files you want to send
- The template to send files from a local machine to a remote host is:
scp file.txt email@example.com:/path/to/destination # What I would run for the example scp script.pbs classifier.py data.csv firstname.lastname@example.org:~/scratch
From Cluster to Personal Computer (Off of Cluster)¶
- Lets say a job had just finished running, and had created a log file with its results.
- To move those result files off of the cluster to your computer, you can use scp.
- For this scenario I will be moving
weather.csv, a csv file that contains the results of a python program I have run on the cluster, to my personal laptop.
- The template to send files from the cluster to a local machine is this:
scp email@example.com:~/path_of_file_to_transfer ~/destination_to_transfer_to #What I would run for the example (transfer the csv off the cluster to a folder on my laptop called "ml") scp firstname.lastname@example.org:~/data/weather.csv ~/ml
- To move a directory, simply use the
-rflag which tells scp to copy the directory and anything contained in it recursively
/scriptsdir From Remote (cluster) to
/PACE-GTon local machine (laptop):
scp -r email@example.com:~/scripts ~/PACE-GT