File transfer using Globus CLI on Tempest (MSU High Performance Computer)

The Globus CLI is standalone software that users can install locally to initiate Globus transfers using a CLI as opposed to the web interface. For installation instructions, see the Globus CLI documentation.

Setup and Login

If you want to transfer files between different Globus endpoints during a job, you can use the Globus CLI tool. To begin, you load the Globus CLI module, log into your Globus account, and grant the proper consent for the Globus CLI to access the collections. This is accomplished with the following code:

module load Globus-CLI         # Loads the latest Globus-CLI version.
globus login                   # Logs into Globus account.
globus session consent ‘urn:globus:auth:scope:transfer.api.globus.org:all[*https:/auth.globus.org/scopes/5485832e-723e-4b52-8472-0410e90902ad/data_access *https:/auth.globus.org/scopes/0dc1297f-9868-4c68-8637-c9b6bd65d3aa/data_access]’

The latter two commands both generate authorization links for you to paste into the web browser of your choice. These will require you to log into your Globus account to generate the authorization codes that can be copied into the terminal. If running the Globus CLI on a personal computer, the software won’t need to be loaded as a module.

Single File Transfers

To begin a file transfer, you need the UUID of the two endpoints that you want to transfer to and from. These UUID’s are found on the web interface overview tab of the collections you want to use. For montana#blackmore, the UUID is “5485832e-723e-4b52-8472-0410e90902ad“. For montana#tempest, the UUID is “0dc1297f-9868-4c68-8637-c9b6bd65d3aa“. You will need to use these values in all of the transfer requests. One easy way to get around typing these out is to create a shell script that exports the UUID’s (and any other UUID’s that are needed) to simple variable names to represent the collections. This allows you to have a setup script for Globus transfers that logs you in and generates the environmental variables. The code for that would be as follows:

#!/bin/bash
module load Globus-CLI
globus login
globus whoami
export blackmore=5485832e-723e-4b52-8472-0410e90902ad
export tempest=0dc1297f-9868-4c68-8637-c9b6bd65d3aa

If you run the code above at the beginning of your job scripts, you will be able to use $blackmore and $tempest in your transfer calls as opposed to manually entering the UUID’s. If you wish to use a personal endpoint as opposed to either Blackmore or Tempest, you can use the UUID of your own collection. 

If you’ve run the setup and then want to transfer a file, you run the following code:

globus transfer $blackmore:/~/path/to/sourceFile.txt $tempest:/~/path/to/destFile.txt

This code will move the files from the listed path on Blackmore to the path on Tempest. However, the batch script will move into the next line of code once the transfer is started. If the next part of the job requires the file to be fully transferred before starting, you can have the job wait until the transfer completes using the following code:

task_id="$(globus transfer $blackmore:/~/path/to/sourceFile.txt $tempest:/~/path/to/destFile.txt --jmespath 'task_id' --format=UNIX)"
globus task wait "$task_id" --timeout 90
if [ $? -eq 0 ]; then
	echo "$task_id completed successfully";
else
	echo "$task_id failed!";
fi

The ‘–timeout’ argument can be modified or removed for longer or shorter timeouts on the data transfers.

Batch File Transfer

If you wish to transfer multiple files in a single request, you can utilize the ‘–batch’ flag in the Globus transfer command. You can either enter the files to transfer manually, or by entering the files in a text file. For the text file example, the layout would look like the following:

~/source/file1.txt ~/dest/file1.txt
~/source/file2.txt ~/dest/file2.txt
~/source/file3.txt ~/dest/file3.txt

Then in your Globus transfer request, you would do a batch transfer while waiting for them all to finish with the following code:

task_id="$(globus transfer $blackmore $tempest --jmespath 'task_id' --format=UNIX --batch batchFile.txt)"
globus task wait "$task_id" --timeout 90
if [ $? -eq 0 ]; then
    echo "$task_id completed successfully";
else
    echo "$task_id failed!";
fi

Full Folder Transfers

If you have a significant amount of files in a folder and just want to transfer the whole folder as opposed to individual files, you can use a recursive transfer. This moves every file in a target folder into the selected destination. The following code would transfer some target folder on Blackmore to a destination folder located on Tempest. Please take note that the destination folder must exist for the transfer to go through. The easiest way to avoid this issue is to have a command that creates the target directory prior to the transfer call

globus transfer $blackmore:$~/path/to/source/ $tempest:~/path/to/dst/ --recursive

This code can be implemented into the same code as above for waiting on the transfer to complete before moving beyond that step of the code.

Full Documentation

Full documentation for the Globus CLI tool can be found at Command Line Interface (CLI) Reference

Example: