Data Transfer Tools
|
The goal of this section is to help you maximize your wide-area network bulk data transfer performance by using file transfer tools that are designed to maximize network throughput. A number of tools are described below. |
Recommended ToolsFor most use cases we recommend using either Globus for reasons given in the background section. |
Sample Results: disk-to-disk testing from Berkeley, CA to Argonne, IL (near Chicago). RTT = 53 ms, network capacity = 10Gbps, RAID = 4 disks, RAID Level-0. Note that to get more than 1 Gbps (125 MB/s) disk to disk requires RAID.
| Tool | Throughput |
| scp | 140 Mbps (17.5 MB/s) |
| HPN patched scp, 1 disk | 760 Mbps (95 MB/s) |
| HPN patched scp, RAID disk | 1.2 Gbps (150 MB/s) |
| Globus, 1 stream, 1 disk | 760 Mbps (95 MB/s) |
| Globus, 1 stream, RAID disk | 1.4 Gbps (175 MB/s) |
| Globus, 4 streams, RAID disk | 5.4 Gbps (675 MB/s) |
| Globus, 8 streams, RAID disk | 6.6 Gbps (825 MB/s) |
Background
Using the right tools, you should be able to move 1 TB between sites in less than 8 hours. Read More
HTTP-TPC
Much of the particle physics community, including the LHC experiments, is replacing GridFTP with HTTP-TPC (third party copy over HTTP). HTTP-TPC uses modern token-based authorization instead of legacy X.509 certificates, and has shown significant performance and scalability for LHC workloads. It has been or is current in the process of being incorporated into multiple tools, such as: … Read More
QUIC (Quick UDP Internet Connections)
QUIC Summary: Deployed for Google services since 2012 Standardized by IETF in RFC 8999, 9000 and 9001 Represents +/- 30% of broadband traffic in 2021 QUIC (Quick UDP Internet Connections, pronounced quick) is an experimental transport layer network protocol designed by Google. The overall goal is to reduce latency compared to that of TCP. Think of QUIC as being similar to… Read More
DOE Site's Data Transfer Nodes
Information on the DTN nodes at the DOE supercomputer centers: NERSC, ALCF, and OLCF.
Read More
