fasterdata.es.netfasterdata.es.netESnet Network Performance Knowledge Base

Data Transfer Tools

The goal of this section is to help you maximize your wide-area network bulk data transfer performance by using file transfer tools that are designed to maximize network throughput. A number of tools are described below.

Recommended Tools

For most use cases we recommend using either Globus Online or GridFTP for reasons given in the background section.

Sample Results: disk-to-disk testing from Berkeley, CA to Argonne, IL (near Chicago). RTT = 53 ms, network capacity = 10Gbps, RAID = 4 disks, RAID Level-0. Note that to get more than 1 Gbps (125 MB/s) disk to disk requires RAID.

Tool Throughput
scp 140 Mbps (17.5 MB/s)
HPN patched scp, 1 disk 760 Mbps (95 MB/s)
HPN patched scp, RAID disk
1.2 Gbps (150 MB/s)
GridFTP, 1 stream, 1 disk
760 Mbps (95 MB/s)
GridFTP, 1 stream, RAID disk
1.4 Gbps (175 MB/s)
GridFTP, 4 streams, RAID disk
5.4 Gbps (675 MB/s)
GridFTP, 8 streams, RAID disk
6.6 Gbps (825 MB/s)

DOE Site's Data Transfer Nodes

Information on the DTN nodes at the DOE supercomputer centers: NERSC, ALCF, and OLCF. Read More »

Say No to scp

Why the standard scp/sftp tools will never work well on a high-speed WAN. Read More »

Case Studies

Improving data movement often means examining a scientific workflow for improvements.  Not all users are comfortable with migration to new and potentially unfamilar tools, which implies that any workflow modification must be done in a manner that is not destructive to overall productivity of a project.  The following case studies highlight different approaches to improving data movement… Read More »