Commercial Tools
TCP Based Tools:
Zettar's zx is a "scale-out" capable data mover solution offering file, object, single-site, and streaming products. zx includes supporting tooling to optimize its site-to-site transfer parameters in terms of network and storage I/O threads, block and piece size, and direct block device access. Network performance is achieved through standard TCP sockets. The zx-File product has demonstrated the ability to saturate a 100G WAN link for disk-to-disk file transfers.
UDP Based Tools:
There are several commercial tools that are based on UDP instead of TCP. For networks with packet loss issues, these tools often perform much better. Some of the tools in this space include:
| Aspera | Aspera FASP is a high-performance reliable transport protocol designed to transfer files and objects over wide area networks with arbitrarily high speeds of up to 10 Gbps on commodity hardware, independent of round-trip delay and packet loss, using a single transport stream. FASP includes an adaptive rate control algorithm that provides congestion avoidance, bandwidth fairness to standard TCP, and disk-based congestion avoidance, and also includes on-the-fly encryption and integrity verification. |
| Data Expedition | Data Expedition developed a set of software tools based on their "Multipurpose Transaction Protocol" (MTP/IP), a UDP-based protocol that uses a data-pull model, and works well on congested links, satellite links, as well as high-speed networks. It is designed to handle files up to 8 exabytes and networks speeds up to 100Gbps and beyond. They also offer SDKs that allow one to integrate MTP-based data transfers into custom applications and some off-the-shelf applications. |
| Tixel | TIXstream from Tixel is another UDP-based solution. |
Note that for UDP-based tools such as these, it is very important to follow the UDP tuning suggestions. For example, this aspera command:
numactl -C 4 /opt/aspera/bin/ascp -S "numactl -C 4 /opt/aspera/bin/ascp -b 1024" -i ~/.ssh/id_rsa -Z 8972 -QTl 20G --no-read --no-write /tmp/test.data 192.168.101.9:/tmp/test.file
Acheived 14 Gbps over a 40G network, compared to only 2Gbps using the default setting for ascp:
ascp -i ~/.ssh/id_rsa -QTl 20G --no-read --no-write /tmp/test.data 192.168.101.9:/tmp/test.file
This 7x difference in performance is typical of UDP-based tools, and is not unique to Aspera's ascp.
