Menu

Data Transfer Tools

The goal of this section is to help you maximize your wide-area network bulk data transfer performance by using file transfer tools that are designed to maximize network throughput. A number of tools are described below.

Recommended Tools

For most use cases we recommend using either Globus Online or GridFTP for reasons given in the background section.

Sample Results: disk-to-disk testing from Berkeley, CA to Argonne, IL (near Chicago). RTT = 53 ms, network capacity = 10Gbps, RAID = 4 disks, RAID Level-0. Note that to get more than 1 Gbps (125 MB/s) disk to disk requires RAID.

Tool Throughput
scp 140 Mbps (17.5 MB/s)
HPN patched scp, 1 disk 760 Mbps (95 MB/s)
HPN patched scp, RAID disk
1.2 Gbps (150 MB/s)
GridFTP, 1 stream, 1 disk
760 Mbps (95 MB/s)
GridFTP, 1 stream, RAID disk
1.4 Gbps (175 MB/s)
GridFTP, 4 streams, RAID disk
5.4 Gbps (675 MB/s)
GridFTP, 8 streams, RAID disk
6.6 Gbps (825 MB/s)

Background

Using the right tools, you should be able to move 1 TB between sites in less than 8 hours.  Read More

Standard Unix Tools

Standard Unix tools besides scp/sftp  Read More

Other Unix Tools

Information on various Unix-based file transfer tools.  Read More

GlobusOnline.jpg

Globus

Secure online service for easy, fast, reliable file transfer.  Read More

GridFTP Quick Start Guide

ssh-based GridFTP Quick Start Guide  Read More

Fast Data Transfer Tool (FDT)

Fast Data Transfer This Java-based tool comes from Caltech. Installation wget http://monalisa.cern.ch/FDT/lib/fdt.jarwget http://monalisa.cern.ch/FDT/scripts/fdtClient.sh Sample Commands start server: java -jar fdt.jar client commands: # memory to memory performance testfdtClient.sh -c hostname -nettest -P 4 & #disk read performance testfdtClient.sh -c hostname -P 2 -pull…  Read More

scp and sftp

Why to avoid use of these tools.  Read More

Commercial Tools

Information on Commercial Tools  Read More

Web Browser Tools/Plugins

Browser plugins for accelerating data downloads.  Read More

MS Windows Tools

Free tools for Windows that support parallel downloading.  Read More