Updated 2023-03-31

Using Rclone to Transfer Files

Overview

  • Rclone is a command line program that allows users to manage files in their cloud storage service
  • Many cloud storage products listed here support Rclone usage
  • Services offered include backing up and encrypting files to cloud storage, restoring and decrypting files from cloud storage, mirroring cloud data to other cloud services or locally, and many other features
  • PACE recommends using Rclone for file transfer when Globus is not an option

When to use Rclone?

  • Globus is typically preferred due to its faster speed, but it cannot connect to the cloud storage providers (e.g. Dropbox, Box, OneDrive/Sharepoint, etc.)
  • Rclone is the next best option for file transfer to/from PACE

Script Download and Installation

  • Note: These directions will use Dropbox as an example but can be completed using other cloud storage services
  • Log into the cluster and create an Rclone directory
  • Download Rclone using the commands:
wget https://downloads.rclone.org/rclone-current-linux-amd64.zip
unzip rclone-current-linux-amd64.zip
  • Launch an Interactive Desktop session with Open OnDemand. This will get a GUI interface on a compute node.
  • In the desktop session, open a Terminal from the Activities tab on the upper left corner
  • Navigate to the directory containing rclone and run the command ./rclone config
  • Follow the default instructions on screen, choosing Dropbox, Box, or whatever is appropriate as the service. For this example, Dropbox will be used.
  • Open Firefox to authenticate

Screenshot

  • Once authentication is complete, a confirmation will appear

Screenshot

  • Go back to the terminal and exit configuration
  • A new remote with the input name is now created
  • Sample output:
Current remotes:

Name                 Type
====                 ====
mydropbox            dropbox

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q

View Configured Remotes

  • Run the command ./rclone listremotes
  • Sample output:
mydropbox:
mybox:
myonedrive:

View Dropbox Content

  • To view all directories in the top level of your dropbox, run ./rclone lsd remote_name:
  • To view all files in your Dropbox, run ./rclone ls remote_name:

Copy Files and Directories to Dropbox

  • To copy a directory from your local system to Dropbox, run the command ./rclone copy ~/file_or_directory_path remote_name:directory_name

Note

If the directory directory_name does not exist in Dropbox, it will be created by Rclone

./rclone lsd mydropbox:
          -1 2023-03-31 16:14:20        -1 rclone_example

./rclone ls mydropbox:
          1359 rclone_example/practice_rclone.png
  • Check the Dropbox contents from the browser to verify if the new directory exists there.

Screenshot

Additional Help

  • For additional directions and help, visit the Rclone list of providers