Globus Online Data Transfer Service

Globus Online addresses the challenges faced by researchers in moving, sharing, and archiving large volumes of data among distributed sites. With Globus Online, you hand-off data movement tasks to a hosted service that manages the entire operation, monitoring performance and errors, retrying failed transfers, correcting problems automatically whenever possible, and reporting status to keep you informed so that you can focus on your research. Command line and web-based interfaces are available. The command line interface, which requires only ssh to be installed on the client, is the method of choice for grid-based workflows.

Getting Started

  1. Register for a Globus Online account.
  2. Sign in to Globus Online using your InCommon/CILogon login. Select the University of Pittsburgh as the Identity Provider and authenticate using your Pitt credentials. The first time you log on, you will be asked to link your CILogon credentials to the Globus account you created in step 1. Subsequent log ons will take you directly to the Globus Dashboard.
  3. Select ‘Start Transfer’ under ‘File Transfer’, or from the drop down menu in the top bar.
  4. In the ‘Start Transfer’ page, you can view the list of available endpoints by clicking the button on the ‘Endpoint’ drop down box.
    • You can use the University of Pittsburgh endpoint “pitt#dtn” as well as other sources or destinations.
    • You can type letters into the box to filter endpoints.
  5. Once you select the “pitt#dtn” endpoint, a login window will pop up. You can access the “pitt#dtn” endpoint on Globus Online by simply using your Pitt username and password. Enter your Pitt username in the ‘Username’ field and Pitt password (not the SSH passphrase) in the ‘Passphrase’ field and click ‘Authenticate’. You can ignore the other fields.
  6. You will see a listing of the contents of your home directory on the University HPC facility. Double click on a directory to view its contents.
  7. Select a file or directory and click on the highlighted ‘arrow button’ to initiate the transfer.

The procedure is same for other endpoints.

Transferring Data Between the University HPC facility and Your Machine

To transfer data between the University HPC facility and your laptop or desktop, you can install Globus Connect on your machine and access it via Globus Online.

  1. Download Globus Connect by going to the Dashboard (select ‘Dashboard’ in the ‘Go To’ drop down box in the top bar) and clicking on ‘Globus Connect’. Globus Connect is available as a one-click install for Mac, Linux, and Windows.
  2. Install Globus Connect following the instructions for your operating system.
  3. Run Globus Connect. When you run Globus Connect for the first time, it will ask you for a setup key.
  4. On the Globus Online web page, go to the Dashboard (select ‘Dashboard’ in the ‘Go To’ drop down box in the top bar), click on ‘Globus Connect’, and enter an endpoint name (eg., mylaptop) to identify your machine on Globus Online.
  5. Click ‘Generate Setup Key’
  6. Copy the setup key and paste it into the Globus Connect window to complete the setup process.

You should now see your machine in the list of endpoints (‘Start Transfer’ screen) identified as <your-username>#<your-globus-connect-name> on Globus Online. You can select it to view the contents of your machine and transfer files to and from it as above.

NOTE:

  • For Windows machines Endpoint use /cygdrive/c/ to see the contents of the C drive (i.e. prefix your directory path with /cygdrive/)
  • Some weird behavior noticed with latest version of IE browser, use Firefox or other browser.

Using the Command Line Interface

In order to use the Globus Online Command Line Interface (CLI), you must add your SSH public key (typically found in $HOME/.ssh/id_rsa.pub) to your Globus Online account.

  1. Select ‘My Account’ from the ‘Go To’ drop down box in the top bar.
  2. Click the ‘Manage Identities’ link in the left column, then click on ‘Add SSH Public Key’.
  3. Paste the contents of your SSH public key into the ‘SSH Public Key’ field (you can optionally enter any name you like in the ‘Alias’ field) and click on ‘Add SSH Key’. It will take a minute or two for the SSH key to become active for use.

You can now log in to the CLI by typing ssh <your-globusonline-username>@cli.globusonline.org at a command prompt. This logs you into a restricted shell where you can execute only Globus Online commands. Type ‘help’ to see all available commands, and help <command-name> or <command-name> -help for help on a specific command. There is also a ‘man’ page for each command accessed as <man <command-name>. For more information on the command line interface, you can refer to the Globus Online CLI Command Reference.

Globus Online Transfer API

Globus Online also has a REST-style interface that facilitates integration into custom applications. More information on the REST API is available at https://transfer.api.globusonline.org/index.html.