fasterdata.es.netfasterdata.es.netESnet Network Performance Knowledge Base

Other Unix Tools

The following are some commonly used open source tools that provide authenticated parallel transfers.

Tool

Summary

Globus Online

Globus Online is from Globus team, and makes it easy to initiate GridFTP transfers from a web browser. For more information see our Quick Start Guide.

 

GridFTP

This tool comes from part of the Globus Toolkit.  To install GridFTP with ssh support, see our Quick Start Guide. The guide also has additional sample commands and other information.

Sample Command

globus-url-copy -p 4 sshftp://data.lbl.gov/home/mydata/myfile \
file://home/mydir/myfile

 

FDT

This Java-based tool comes from Caltech. It is much easier to install and use than GridFTP, but is not as fast in our tests, and the error handling is not great.

Installation

wget http://monalisa.cern.ch/FDT/lib/fdt.jar
wget http://monalisa.cern.ch/FDT/scripts/fdtClient.sh

Sample Commands

server: java -jar fdt.jar
client: fdtClient.sh -c hostname /path/to/files -P 8 -d /path/to/outputDir

UDR

UDR is a replacement for rsync that used the UDP-based UDT protocol instead of TCP. It works very well for speeds up to 1-2 Gbps without tuning, and can go faster with proper tuning.

Sample Command

udr rsync -av --stats --progress /home/user/tmp/ hostname.com:/home/user/tmp

bbftp

This tool comes from the Babar Project.

Sample Command

bbftp -p 4 -e 'put myfile' -E '/usr/local/bin/bbftpd -s' remotehost
 

bbcp

This tool was developed at SLAC for Baber. More info on using bbcp is available from Caltech.

Sample Command

bbcp -P 4 -v myfile remotehost:filename
 

nuttscp

This is a simple perl script wrapper that uses ssh and the nuttcp tool to copy files, and can achieve very high throughput.

Sample Command

nuttscp -v -N 4 -l 256K -f /mydir/myfile \
remotehost:/data1/mydir/myfile.out