Data Transfer Tools
|
The goal of this section is to help you maximize your wide-area network bulk data transfer performance by using file transfer tools that are designed to maximize network throughput. A number of tools are described below. |
Recommended ToolsFor most use cases we recommend using Globus for reasons given in the background section. |
Background
Using the right tools, you should be able to move 1 TB between sites in less than 8 hours. Read More
Say No to scp/sftp
Why the standard scp/sftp tools will never work well on a high-speed WAN. Read More
HPN-SSH
HPN-SSH Description HPN-SSH is a drop in replacement for sftp/scp that removes the internal buffering restrictions of standard sftp/scp, providing over 100X speedup on a high RTT WAN path. The new version of HPN-SSH is very easy to install. For best performance it must be installed on both client and server sides, and may involve modifying host or router firewall rules. However server side only… Read More
HTTP-TPC
Much of the particle physics community, including the LHC experiments, is replacing GridFTP with HTTP-TPC (third party copy over HTTP). HTTP-TPC uses modern token-based authorization instead of legacy X.509 certificates, and has shown significant performance and scalability for LHC workloads. It has been or is current in the process of being incorporated into multiple tools, such as: … Read More
QUIC (Quick UDP Internet Connections)
QUIC (Quick UDP Internet Connections, pronounced “quick”) is a modern transport layer network protocol originally developed by Google and now standardized by the IETF. Its primary goal is to reduce latency compared to traditional TCP, especially during connection setup and in scenarios involving packet loss. QUIC can be thought of as a combination of TCP + TLS, implemented over UDP. Unlike… Read More
DOE Site's Data Transfer Nodes
Information on the DTN nodes at the DOE supercomputer centers: NERSC, ALCF, and OLCF.
Read More
