Updated 2020-04-23

File Transfer on testflight-coda

Overview

  • File transfer to testflight-coda from another Cluster is different than the traditional file transfer methods
  • Your existing data directories will be mounted in read-only mode on the testflight-coda headnode only (NOT on compute nodes)
  • You should copy all files you need to use into a separate folder to test on tf-coda

Important

testflight-coda storage volumes are only temporary and will be available during the testing period, so any data or software critical to your research should also remain on existing resources in the Rich datacenter

Making Files Accessible on testflight-coda

  • Log into the headnode for the Cluster you normally use (such as login-s)
  • Identify the data which you will need for testing and development on the testflight-coda system and copy them into the ~/data folder with the cp command
  • As an example, if you want make the files in ~/test/tf-coda available for access on testflight-coda
cp -r ~/test/tf-coda ~/data
  • The files you put in ~/data are now available for access in read-only mode on testflight-coda

Accessing the Files on testflight-coda

  • Log into testflight-coda through login-testflight-coda
  • Run pace-whoami and take a look at the line that says Your project directory (~/data)
  • If you take the filepath on this line and replace /gpfs with /storage/rich, you can access the data directory with the full path name of the symlink
  • Ex: cd /storage/rich/pace2/project/pf1/<username>

Working With Files on testflight-coda

Warning

Your data directory files are read-only and only accessible from the login node. This means that you CANNOT use them to run jobs as they currently are. Instead, please make a copy of them if you wish to use them as per the instructions in this section.

  • After accessing the mounted read-only data directory from Rich, you can make a copy of the files you need with the cp command to create copies you can work with on compute nodes or write to

Important

Please only copy what is needed for testing and NOT EVERYTHING that is in your data directory. If you need to copy more than 50GB, please contact your assigned PACE research scientist.

  • For example, to create a copy of the tf-coda directory in the mounted data directory and put it in testflight-coda's ~/data directory, you can run:
cp -r /storage/rich/pace2/project/pf1/<username>/tf-coda ~/data
  • Please copy the files to the ~/data directory. The home directory has a limited quota

Important

Again, you will need to do this for any file you plan to use on a compute node (even ones you are only reading from) as the mounted data directory will only be accessible from the headnode

Transfer Files off of testflight-coda

Note

Files on testflight-coda are NOT backed up. Please keep a copy for yourself of any key scripts, codes, or executables you develop.

  • File transfer off of testflight-coda must be done through either SCP or GitHub since there is no Globus endpoint set up

Transfer with SCP on Linux/Mac

  • To transfer files from testflight-coda to your local machine, you would run the following command with the necessary changes on your local machine:
scp <username>@login-testflight-coda.pace.gatech.edu:~/path_of_file_to_transfer ~/destination_to_transfer_to

Note

To move a directory, simply use the -r flag which tells scp to copy the directory and anything contained in it recursively

Transfer with SCP on Windows

  • Please see our WinSCP Guide for instructions on how to get WinSCP set up if you haven't already
  • In the hostname box, use the address for your account on testflight-coda such as <username>@login-testflight-coda.pace.gatech.edu

Transfer with Git

  • Load the Git module on testflight-coda with module load git
  • Utilize Git as you normally do