Data Transfer Tools
The goal of this section is to help you maximize your wide-area network bulk data transfer performance by using file transfer tools that are designed to maximize network throughput. A number of tools are described below.
Sample Results: disk-to-disk testing from Berkeley, CA to Argonne, IL (near Chicago). RTT = 53 ms, network capacity = 10Gbps, RAID = 4 disks, RAID Level-0. Note that to get more than 1 Gbps (125 MB/s) disk to disk requires RAID.
|scp||140 Mbps (17.5 MB/s)|
|HPN patched scp, 1 disk||760 Mbps (95 MB/s)|
|HPN patched scp, RAID disk||1.2 Gbps (150 MB/s)|
|GridFTP, 1 stream, 1 disk||760 Mbps (95 MB/s)|
|GridFTP, 1 stream, RAID disk||1.4 Gbps (175 MB/s)|
|GridFTP, 4 streams, RAID disk||5.4 Gbps (675 MB/s)|
|GridFTP, 8 streams, RAID disk||6.6 Gbps (825 MB/s)|
QUIC (Quick UDP Internet Connections, pronounced quick) is an experimental transport layer network protocol designed by Google. The overall goal is to reduce latency compared to that of TCP. Think of QUIC as being similar to TCP+TLS+HTTP/2 implemented on UDP. Because TCP is implemented at the lowest levels of machinery (operating systems, routing firmware), making changes to TCP is next to… Read More
Improving data movement often means examining a scientific workflow for improvements. Not all users are comfortable with migration to new and potentially unfamilar tools, which implies that any workflow modification must be done in a manner that is not destructive to overall productivity of a project. The following case studies highlight different approaches to improving data movement… Read More