Menu

Data Transfer Tools

The goal of this section is to help you maximize your wide-area network bulk data transfer performance by using file transfer tools that are designed to maximize network throughput. A number of tools are described below.

Recommended Tools

For most use cases we recommend using either Globus Online or GridFTP for reasons given in the background section.

Sample Results: disk-to-disk testing from Berkeley, CA to Argonne, IL (near Chicago). RTT = 53 ms, network capacity = 10Gbps, RAID = 4 disks, RAID Level-0. Note that to get more than 1 Gbps (125 MB/s) disk to disk requires RAID.

Tool Throughput
scp 140 Mbps (17.5 MB/s)
HPN patched scp, 1 disk 760 Mbps (95 MB/s)
HPN patched scp, RAID disk
1.2 Gbps (150 MB/s)
GridFTP, 1 stream, 1 disk
760 Mbps (95 MB/s)
GridFTP, 1 stream, RAID disk
1.4 Gbps (175 MB/s)
GridFTP, 4 streams, RAID disk
5.4 Gbps (675 MB/s)
GridFTP, 8 streams, RAID disk
6.6 Gbps (825 MB/s)

Background

Using the right tools, you should be able to move 1 TB between sites in less than 8 hours.  Read More

Standard Unix Tools

Standard Unix tools besides scp/sftp  Read More

Other Unix Tools

Information on various Unix-based file transfer tools.  Read More

GlobusOnline.jpg

Globus

Secure online service for easy, fast, reliable file transfer.  Read More

GridFTP Quick Start Guide

ssh-based GridFTP Quick Start Guide  Read More

scp and sftp

Why to avoid use of these tools.  Read More

Commercial Tools

Information on Commercial Tools  Read More

Web Browser Tools/Plugins

Browser plugins for accelerating data downloads.  Read More

MS Windows Tools

Free tools for Windows that support parallel downloading.  Read More

Special Purpose Tools

Tools to access HPSS systems  Read More