Transferring Files

Frank supports SCP (Secure Copy) protocol to securely transfer files in and out of the system. There are various SCP programs which you may want to use depending on your client OS .

Command-line scp program

Most Linux based OSs have the OpenSSH suite installed. Also available on Windows through Cygwin. The suite provides the commandline utility scp, which works the same as the regular copy program cp. The remote target is specified using username@frank.sam.pitt.edu: syntax. username can be omitted if the user name in the client is the same as in Frank. Or, recommended, a shorthand can be used, e.g. frank: where frank is an alias (can be any name) defined in ~/.ssh/config. These aliases can be used for both scp and ssh. See the man page for ssh_config.

To copy a file using scp:

# Upload a file named foo.txt to home directory on Frank 
scp foo.txt username@frank.sam.pitt.edu:
 
# Or using the alias (if defined in `~/.ssh/config`)
scp foo.txt frank:
 
# A whole directory can be copied as
scp -r foo/bar/ frank: 
 
# Download foo.txt at home directory on Frank to local directory
scp frank:foo.txt . 

The scp man page for more details.

rsync

rsync is a software application for Unix, Linux and Windows systems which synchronizes files and directories from one location to another while minimizing data transfer using delta encoding when appropriate. rsync can copy or display directory contents and copy files, optionally using compression and recursion. It is popular utility and comes installed in most Linux distributions. rsync is the recommended way to transfer files in and out of Frank.

rsync works for local transfers as well as remote transfers using the SCP protocol and a syntax similar to the scp program:

# Transfer the contents of a directory to another location
#   -a is similar to -r in scp, but preserves symlinks,
#   permissions, ownership, etc. 
rsync -a /my/output/ ~/backup
 
# If the trailing slash `/` is omitted from the source location
# then the directory and the contents will be transferred.
# I.e. the following will create a directory `output' under 
# `~/backup`, with all the content of the original `output`.
rsync -a /my/output ~/backup
 
# Remote targets are also supported.
# This copies directory `output` and its contents to 
# the home directory `~/output` on Frank. 
rsync -a /my/output username@frank.sam.pitt.edu:
# or using the alias (see above for scp)
rsync -a /my/output frank: 
 
# Broken transfers can be resumed with the -P or --partial flags. 
# The -P flag also turns on a progress display for each file
# transferred.
rsync -aP /my/output frank: 

rsync has many more options such as transfer exclusion lists, compression, backup, transfer modes, etc. Refer to man pages for more information.

WinSCP

WinSCP is a graphical based utility for Windows systems. It is versatile, easy to use, and free. It supports both SCP and SFTP protocols, and resumes broken transfers. Strongly recommended for Windows users. You can download WinSCP at http://winscp.net/eng/download.php.

Globus Online, GridFTP

Globus Online addresses the challenges faced by researchers in moving, sharing, and archiving large volumes of data among distributed sites. With Globus Online, you hand-off data movement tasks to a hosted service that manages the entire operation, monitoring performance and errors, retrying failed transfers, correcting problems automatically whenever possible, and reporting status to keep you informed so that you can focus on your research. Command line and web-based interfaces are available. The command line interface, which requires only ssh to be installed on the client, is the method of choice for grid-based workflows.

Details: http://core.sam.pitt.edu/globusonline