Updated 2020-01-14

Using globus-url-copy command line tool

Mac / Linux

You will want to do the first part of the globus-toolkit install as outlined at Install Globus Connect Server namely:

sudo curl -LOs https://downloads.globus.org/toolkit/globus-connect-server/globus-connect-server-repo-latest.noarch.rpm
sudo rpm --import https://downloads.globus.org/toolkit/gt6/stable/repo/rpm/RPM-GPG-KEY-Globus
sudo yum install globus-connect-server-repo-latest.noarch.rpm
#Install EPEL repo
sudo curl -LOs https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
sudo yum install epel-release-latest-7.noarch.rpm

and then the specific globus-url-copy and myproxy software

yum install tcllib
yum install globus-gass-copy-progs
yum install myproxy
yum install globus-proxy-utils

You may also want to setup globusconnectpersonal, for managing a globus connect personal endpoint that you can access via the globus.org portal or via the globus-cli. It is not necessary for using globus-url-copy to upload files and directories to a remote endpoint.

wget https://downloads.globus.org/globus-connect-personal/linux/stable/globusconnectpersonal-latest.tgz
tar xzf globusconnectpersonal-latest.tgz
cd globusconnectpersonal-x.y.z
# setup globus connect personal in ~/.globusonline/lta, or pass -dir ~/.somethingelse to setup in ~/.somethingelse
./globusconnect -setup <whatever key you got from globus-cli or create endpoint on globus portal> # -dir ~/.somethingelse
#add -restrict-paths if you want to add additional shared paths, or edit ~/.globusonline/lta/config-paths
./globusconnect -start -restrict-paths RW~/,RW/writable/directory,R/read/only/directory,N/none/accessible/directory &
 # -dir ~/.somethingelse 

Log in to a local globus endpoint such as globus-research.pace.gatech.edu or iw-dm-4.pace.gatech.edu. In this example I use the PACE Research internal high speed endpoint connected to the VAPOR networks for servers connected to scientific instruments that are connected to the VAPOR network. The argument -t 8760 sets the proxy life time for 8760 hours or a year. The -b argument bootstraps the certificate directory from the endpoint (also implies -T trustroots from that server) and -s argument is the hostname of the server.

[amcneill3@rich116-f39-15 globusconnectpersonal-2.3.6]$ myproxy-logon -t 8760 -b -T -s globus-research
Bootstrapping MyProxy server root of trust.
New trusted MyProxy server: /C=US/O=Globus Consortium/OU=Globus Connect Service/CN=2bfacbe2-2af2-11e9-9fa4-0a06afd4a22e
New trusted CA (a059cd44.0): /C=US/O=Globus Consortium/CN=Globus Connect CA 3
Server authorization failed.  Server identity does not match expected identity.
If the server identity is acceptable, set
MYPROXY_SERVER_DN="/C=US/O=Globus Consortium/OU=Globus Connect Service/CN=2bfacbe2-2af2-11e9-9fa4-0a06afd4a22e"
and try again.
`[amcneill3@rich116-f39-15 globusconnectpersonal-2.3.6]$ export MYPROXY_SERVER_DN="/C=US/O=Globus Consortium/OU=Globus Connect Service/CN=2bfacbe2-2af2-11e9-9fa4-0a06afd4a22e"
[amcneill3@rich116-f39-15 globusconnectpersonal-2.3.6]$ myproxy-logon -t 8760 -b -T -s globus-research
Enter MyProxy pass phrase:
A credential has been received for user amcneill3 in /tmp/x509up_u296017.
Trust roots have been installed in /nv/hp1/amcneill3/.globus/certificates/.

It is a good idead to same the environmental variable MYPROXY_SERVER_DN for future use, by placing it in your .bashrc or .bash_profile file as:

export MYPROXY_SERVER_DN="/C=US/O=Globus Consortium/OU=Globus Connect Service/CN=2bfacbe2-2af2-11e9-9fa4-0a06afd4a22e

To list the contents of a remote directory using globus-url-copy, (-ss sets source DN, we want to set this to the DN for globus-research, otherwise, it will fail and complain that the hostname globus-research does not match the DN) do:

[amcneill3@rich116-f39-15 globus]$ globus-url-copy -ss "$MYPROXY_SERVER_DN" -list gsiftp://globus-research/~/scratch/5gbzd/

To upload a single file using globus-url-copy (here we want to set destination DN with -ds), do:

[amcneill3@rich116-f39-15 globus]$ globus-url-copy -ds "$MYPROXY_SERVER_DN" -fast -rst -rst-retries 0 -tcp-bs 1G -p 16 1G.dat gsiftp://globus-research/~/scratch/1Gb.dat
  • -fast = optimize transfer
  • -rst = reset if there is an interruption
  • -rst-restries 0 = retry an infinite amount of times if there are resets
  • -tcp-bs = set buffer size for transfer cache
  • -p 16 = set parrallesim (can be 1 to 16)
  • 1G.dat = source file
  • gsiftp://globus-research/~/scratch/1Gb.dat = destination file on globus-server "PACE Research" endpoint

In order to upload a directory using globus-url-copy, use the -r (recursive) and -cd (create destination) arguments

[amcneill3@rich116-f39-15 globus]$ globus-url-copy -ds "$MYPROXY_SERVER_DN" -fast -rst -rst-retries 0 -tcp-bs 1G -p 16 -r -cd 5gbz/ gsiftp://globus-research/~/scratch/5gbzd/

If you want to verify-checksum of the transfer then add the -verify-checksum argument

[amcneill3@rich116-f39-15 globus]$ globus-url-copy -ds "$MYPROXY_SERVER_DN" -verify-checksum -fast -rst -rst-retries 0 -tcp-bs 1G -p 16 -r -cd 5gbz/ gsiftp://globus-research/~/scratch/5gbzd/

Windows

Install python (in our case we install python 3.8.0 amd64 including cli and pip) Screenshot

Open up the windows command prompt tool and verify python is in your path

C:\Users\amcneill3>setx PATH "C:\Users\amcneill3\AppData\Roaming\Python\Python38\Scripts;%PATH%"

Use pip to install globus-cli

C:\Users\amcneill3>pip install --user globus-cli
Collecting globus-cli
  Using cached https://files.pythonhosted.org/packages/46/45/6e9fbb01549f05715591fcc3835eba11487b58e884479efba6e10eb274e7/globus-cli-1.11.0.tar.gz
Collecting globus-sdk==1.8.0 (from globus-cli)
  Using cached https://files.pythonhosted.org/packages/be/c1/ccf8e8ebeb229887e6355e87e9cc9f07e06b1661b3c6e50311566bae9f28/globus_sdk-1.8.0-py2.py3-none-any.whl
Collecting click<8.0,>=7.0 (from globus-cli)
  Using cached https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl
Collecting jmespath==0.9.4 (from globus-cli)
  Using cached https://files.pythonhosted.org/packages/83/94/7179c3832a6d45b266ddb2aac329e101367fbdb11f425f13771d27f225bb/jmespath-0.9.4-py2.py3-none-any.whl
Collecting configobj<6.0.0,>=5.0.6 (from globus-cli)
  Using cached https://files.pythonhosted.org/packages/64/61/079eb60459c44929e684fa7d9e2fdca403f67d64dd9dbac27296be2e0fab/configobj-5.0.6.tar.gz
Collecting requests<3.0.0,>=2.0.0 (from globus-cli)
  Using cached https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
Collecting six<2.0.0,>=1.10.0 (from globus-cli)
  Using cached https://files.pythonhosted.org/packages/65/26/32b8464df2a97e6dd1b656ed26b2c194606c16fe163c695a992b36c11cdf/six-1.13.0-py2.py3-none-any.whl
Collecting cryptography<3.0.0,>=1.8.1 (from globus-cli)
  Using cached https://files.pythonhosted.org/packages/27/da/8ee8775f0b6883b6c53e3f2d8a9a1e109661176446ec09f27c070dfc7baf/cryptography-2.8-cp38-cp38-win_amd64.whl
Collecting pyjwt[crypto]<2.0.0,>=1.5.3 (from globus-sdk==1.8.0->globus-cli)
  Using cached https://files.pythonhosted.org/packages/87/8b/6a9f14b5f781697e51259d81657e6048fd31a113229cf346880bb7545565/PyJWT-1.7.1-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5 (from requests<3.0.0,>=2.0.0->globus-cli)
  Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests<3.0.0,>=2.0.0->globus-cli)
  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests<3.0.0,>=2.0.0->globus-cli)
  Using cached https://files.pythonhosted.org/packages/b9/63/df50cac98ea0d5b006c55a399c3bf1db9da7b5a24de7890bc9cfd5dd9e99/certifi-2019.11.28-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests<3.0.0,>=2.0.0->globus-cli)
  Using cached https://files.pythonhosted.org/packages/b4/40/a9837291310ee1ccc242ceb6ebfd9eb21539649f193a7c8c86ba15b98539/urllib3-1.25.7-py2.py3-none-any.whl
Collecting cffi!=1.11.3,>=1.8 (from cryptography<3.0.0,>=1.8.1->globus-cli)
  Using cached https://files.pythonhosted.org/packages/4f/c9/135197be2a44f8dbe59373a55018843aa586e14af23e813897f40b6689f6/cffi-1.13.2-cp38-cp38-win_amd64.whl
Collecting pycparser (from cffi!=1.11.3,>=1.8->cryptography<3.0.0,>=1.8.1->globus-cli)
  Using cached https://files.pythonhosted.org/packages/68/9e/49196946aee219aead1290e00d1e7fdeab8567783e83e1b9ab5585e6206a/pycparser-2.19.tar.gz
Installing collected packages: six, pycparser, cffi, cryptography, pyjwt, idna, chardet, certifi, urllib3, requests, globus-sdk, click, jmespath, configobj, globus-cli
  Running setup.py install for pycparser ... done
  Running setup.py install for configobj ... done
  Running setup.py install for globus-cli ... done
Successfully installed certifi-2019.11.28 cffi-1.13.2 chardet-3.0.4 click-7.0 configobj-5.0.6 cryptography-2.8 globus-cli-1.11.0 globus-sdk-1.8.0 idna-2.8 jmespath-0.9.4 pycparser-2.19 pyjwt-1.7.1 requests-2.22.0 six-1.13.0 urllib3-1.25.7
WARNING: You are using pip version 19.2.3, however version 19.3.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

Verify that you can run the globus command:

C:\Users\amcneill3>globus --version
Installed Version: 1.11.0
Latest Version:    1.11.0
You are running the latest version of the Globus CLI

Before you can utilize the globus cli effectively you will have to log in. Here is a way you can start the login, get a url, and use the url to use your CILogin/Gatech credentials to login and start a cli session on your personal node:

C:\Users\amcneill3>globus endpoint search PACE-Hive
Globus CLI Error: No Authentication provided. Make sure you have logged in with 'globus login'.
C:\Users\amcneill3>globus login --no-local-server
Please authenticate with Globus here:
------------------------------------
https://auth.globus.org/v2/oauth2/authorize?client_id=4629a0d6-bd3f-4dd0-8a63-43d0b44a0429&redirect_uri=https%3A%2F%2Fauth.globus.org%2Fv2%2Fweb%2Fauth-code&scope=openid+profile+email+urn%3Aglobus%3Aauth%3Ascope%3Aauth.globus.org%3Aview_identity_set+urn%3Aglobus%3Aauth%3Ascope%3Atransfer.api.globus.org%3Aall&state=_default&response_type=code&access_type=offline&prompt=login
------------------------------------
Enter the resulting Authorization Code here: mkcbG49wErehnKGxiNjC2DypKSUGLh
You have successfully logged in to the Globus CLI!
You can check your primary identity with
  globus whoami
For information on which of your identities are in session use
  globus session show
Logout of the Globus CLI with
  globus logout

You should now be able to search for your own globus connect personal endpoint, and an endpoint at GaTech you want to connect to, and save the ID for each endpoint to environment variables you can use later to run query and transfer commands. Here we set the source to SOR and destination to DES

C:\Users\amcneill3>globus endpoint search pace-windows-dev01
ID                                   | Owner                | Display Name
------------------------------------ | -------------------- | ------------------
c528a22e-1d38-11ea-9705-021304b0cca7 | amcneill3@gatech.edu | pace-windows-dev01
C:\Users\amcneill3>globus endpoint search PACE-Hive
ID                                   | Owner                   | Display Name
------------------------------------ | ----------------------- | ------------
c2c0b96a-ca9d-11e9-939b-02ff96a5aa76 | gatechpace@globusid.org | PACE Hive
C:\Users\amcneill3>set  SOR="c528a22e-1d38-11ea-9705-021304b0cca7"
C:\Users\amcneill3>set DES="c2c0b96a-ca9d-11e9-939b-02ff96a5aa76"

Do an ls at each endpoint to view files or directories:

C:\Users\amcneill3>globus ls %SOR%
My Music/
My Pictures/
My Videos/
stuff/
50MB.zip
desktop.ini

C:\Users\amcneill3>globus ls %DES%:scratch/
50GB-in-medium-files/
100M.dat
10Gc.dat
10Gd.dat
4.2.0.2.tar.gz
817787_001_spp_2015.04.0-SPP2015040.2015_0407.5.iso

Transfer a single file from your source to your destination:

C:\Users\amcneill3>globus transfer %SOR%:50MB.zip  %DES%:scratch/50MB.zip
Message: The transfer has been accepted and a task has been created and queued for execution
Task ID: 9269e85a-1d3f-11ea-9706-021304b0cca7
C:\Users\amcneill3>globus task list
Task ID                              | Status    | Type     | Source Display Name | Dest Display Name | Label
------------------------------------ | --------- | -------- | ------------------- | ----------------- | -----
9269e85a-1d3f-11ea-9706-021304b0cca7 | SUCCEEDED | TRANSFER | pace-windows-dev01  | PACE Hive         | NULL
885e8992-1d3f-11ea-ab42-0a7959ea6081 | ACTIVE    | TRANSFER | pace-windows-dev01  | PACE Hive         | NULL
61179284-1d3f-11ea-9706-021304b0cca7 | ACTIVE    | TRANSFER | pace-windows-dev01  | PACE Hive         | NULL
430e8f7c-1d3f-11ea-9706-021304b0cca7 | SUCCEEDED | DELETE   | PACE Hive           | NULL              | NULL
21ecd72c-1d3f-11ea-b955-0e16720bb42f | SUCCEEDED | TRANSFER | pace-windows-dev01  | PACE Hive         | NULL
e2a315c8-1d3d-11ea-9706-021304b0cca7 | SUCCEEDED | DELETE   | PACE Hive           | NULL              | NULL
bfffed2a-1d3d-11ea-9706-021304b0cca7 | SUCCEEDED | TRANSFER | pace-windows-dev01  | PACE Hive         | NULL
53ef1066-1d3d-11ea-b955-0e16720bb42f | ACTIVE    | TRANSFER | pace-windows-dev01  | PACE Hive         | NULL
21a6bcbc-1d3d-11ea-ab42-0a7959ea6081 | ACTIVE    | TRANSFER | pace-windows-dev01  | PACE Hive         | NULL
e08c2fbe-1d3c-11ea-9705-021304b0cca7 | ACTIVE    | TRANSFER | pace-windows-dev01  | PACE Hive         | NULL
C:\Users\amcneill3>globus ls %DES%:scratch/
50GB-in-medium-files/
100M.dat
10Gc.dat
10Gd.dat
4.2.0.2.tar.gz
50MB.zip
817787_001_spp_2015.04.0-SPP2015040.2015_0407.5.iso

Transfer a directory from your source to your destination:

C:\Users\amcneill3>globus ls %DES%:scratch/
50GB-in-medium-files/
100M.dat
10Gc.dat
10Gd.dat
4.2.0.2.tar.gz
50MB.zip
817787_001_spp_2015.04.0-SPP2015040.2015_0407.5.iso
C:\Users\amcneill3>globus ls %SOR%:stuff/
100MB.zip
200MB.zip
C:\Users\amcneill3>globus transfer --recursive %SOR%:stuff/ %DES%:scratch/
Message: The transfer has been accepted and a task has been created and queued for execution
Task ID: d29f0dde-1d43-11ea-ab42-0a7959ea6081
C:\Users\amcneill3>globus task list
Task ID                              | Status    | Type     | Source Display Name | Dest Display Name | Label
------------------------------------ | --------- | -------- | ------------------- | ----------------- | -----
d29f0dde-1d43-11ea-ab42-0a7959ea6081 | SUCCEEDED | TRANSFER | pace-windows-dev01  | PACE Hive         | NULL
b2a8ab3e-1d43-11ea-ab42-0a7959ea6081 | SUCCEEDED | TRANSFER | pace-windows-dev01  | PACE Hive         | NULL
639be354-1d42-11ea-9706-021304b0cca7 | SUCCEEDED | DELETE   | PACE Hive           | NULL              | NULL
9269e85a-1d3f-11ea-9706-021304b0cca7 | SUCCEEDED | TRANSFER | pace-windows-dev01  | PACE Hive         | NULL
885e8992-1d3f-11ea-ab42-0a7959ea6081 | FAILED    | TRANSFER | pace-windows-dev01  | PACE Hive         | NULL
61179284-1d3f-11ea-9706-021304b0cca7 | FAILED    | TRANSFER | pace-windows-dev01  | PACE Hive         | NULL
430e8f7c-1d3f-11ea-9706-021304b0cca7 | SUCCEEDED | DELETE   | PACE Hive           | NULL              | NULL
21ecd72c-1d3f-11ea-b955-0e16720bb42f | SUCCEEDED | TRANSFER | pace-windows-dev01  | PACE Hive         | NULL
e2a315c8-1d3d-11ea-9706-021304b0cca7 | SUCCEEDED | DELETE   | PACE Hive           | NULL              | NULL
bfffed2a-1d3d-11ea-9706-021304b0cca7 | SUCCEEDED | TRANSFER | pace-windows-dev01  | PACE Hive         | NULL
C:\Users\amcneill3>globus ls %DES%:scratch/
50GB-in-medium-files/
100M.dat
100MB.zip
10Gc.dat
10Gd.dat
200MB.zip
4.2.0.2.tar.gz
50MB.zip
817787_001_spp_2015.04.0-SPP2015040.2015_0407.5.iso

You can add or enforce other arguments to your transfer, such as -verify-checksum for quality of service when sending transfers to make sure what you sent is what is actually at your source. Here is a list of the help for the "globus transfer" command. You can get similar output for help for the other globus cli commands.

C:\Users\amcneill3>globus transfer -h
Usage: globus transfer [OPTIONS] SOURCE_ENDPOINT_ID[:SOURCE_PATH]
                       DEST_ENDPOINT_ID[:DEST_PATH]
  Copy a file or directory from one endpoint to another as an asynchronous
  task.
  Batched Input
  ===
  If you use `SOURCE_PATH` and `DEST_PATH` without the `--batch` flag, you
  will submit a single-file or single-directory transfer task. This has
  behavior similar to `cp` and `cp -r`, across endpoints of course.
  Using `--batch`, `globus transfer` can submit a task which transfers
  multiple files or directories. Paths to transfer are taken from stdin.
  Lines are split on spaces, respecting quotes, and every line is treated as
  a file or directory to transfer.
  Lines are of the form
  [--recursive] SOURCE_PATH DEST_PATH
  Skips empty lines and allows comments beginning with "#".
  If you use `--batch` and a commandline SOURCE_PATH and/or DEST_PATH, these
  paths will be used as dir prefixes to any paths on stdin.
  Sync Levels
  ===
  Sync Levels are ways to decide whether or not files are copied, with the
  following definitions:
  EXISTS: Determine whether or not to transfer based on file existence. If
  the destination file is absent, do the transfer.
  SIZE: Determine whether or not to transfer based on the size of the file.
  If destination file size does not match the source, do the transfer.
  MTIME: Determine whether or not to transfer based on modification times.
  If source has a newer modififed time than the destination, do the
  transfer.
  CHECKSUM: Determine whether or not to transfer based on checksums of file
  contents. If source and destination contents differ, as determined by a
  checksum of their contents, do the transfer.
  If a transfer fails, CHECKSUM must be used to restart the transfer. All
  other levels can lead to data corruption.
Options:
  --map-http-status TEXT          Map HTTP statuses to any of these exit
                                  codes: 0,1,50-99. e.g. "404=50,403=51"
  --jmespath, --jq TEXT           A JMESPath expression to apply to json
                                  output. Takes precedence over any specified
                                  '--format' and forces the format to be json
                                  processed by this expression
  -F, --format [unix|json|text]   Output format for stdout. Defaults to text
  -h, --help                      Show this message and exit.
  -v, --verbose                   Control level of output
  --skip-activation-check         Submit the task even if the endpoint(s)
                                  aren't currently activated.
  --deadline [%Y-%m-%d|%Y-%m-%dT%H:%M:%S|%Y-%m-%d %H:%M:%S]
                                  Set a deadline for this to be canceled if
                                  not completed by.
  --label TEXT                    Set a label for this task.
  --submission-id TEXT            Task submission ID, as generated by `globus
                                  task generate-submission-id`. Used for safe
                                  resubmission in the presence of network
                                  failures.
  --notify TEXT                   Comma separated list of task events which
                                  notify by email. 'on' and 'off' may be used
                                  to enable or disable notifications for all
                                  event types. Otherwise, use 'succeeded',
                                  'failed', or 'inactive'
  --dry-run                       Don't actually submit the task, print
                                  submission data instead
  -r, --recursive                 SOURCE_PATH and DEST_PATH are both
                                  directories, do a recursive dir transfer
  -s, --sync-level [exists|size|mtime|checksum]
                                  How will the Transfer task determine whether
                                  or not to actually transfer a file over the
                                  network?
  --preserve-mtime                Preserve file and directory modification
                                  times.
  --verify-checksum / --no-verify-checksum
                                  Verify checksum after transfer.  [default:
                                  True]
  --encrypt                       Encrypt data sent through the network.
  --delete                        Delete extraneous files in the destination
                                  directory. Only applies to recursive
                                  directory transfers.
  --batch                         Accept a batch of source/dest path pairs on
                                  stdin (i.e. run in batchmode). Uses
                                  SOURCE_ENDPOINT_ID and DEST_ENDPOINT_ID as
                                  passed on the commandline. Commandline paths
                                  are still allowed and are used as prefixes
                                  to the batchmode inputs.

(to be completed)