Data Transfer Node Reference Implementation
ESnet has assembled a reference implementation of a host that can be deployed as a DTN or as a high-speed GridFTP test machine. The host can easily fill a 10Gbps network connection with disk-to-disk data transfers using GridFTP, and has demonstrated memory to memory transfers of 39.5 Gbps TCP using a 40G NIC.
The total cost of this server was around $12K in 2014, which included 16 250G SATA-3 HDD drives, which can sustain 2.2GByte/sec (17.7Gbps) reading from disk . Depending on your DTN capacity and disk IO requirements, the cost could be considerably more. Our server is 2U but you might want to consider getting a 4U to give you more room for future expansion.
- Motherboard: SuperMicro X9DRi-F
- CPU: 2 x Intel(R) Xeon Ivy Bridge E5-2643V2 3.5GHz 6 Cores (Total 12 Cores)
- Memory: 96G ( (12) 8GB DDR3-1866MHz RAM ECC/REG )
- RAID: Adaptec ASR-81605ZQ (16 ports)
- 16 x Western Digital, model WD2500BHTZ-0
- Network Controller:
- 10G: Intel X520, Myricom 10G-PCIE2-8C2-2S, Chelsio T5
- 40G: Mellanox MCX312A-XCBT
- 100G: Mellanox MCX455A-ECAT
We use the most recent CentOS-7 distribution of Linux, and have configured the data drives as RAID6.
Detailed Tuning Information is here.
Performance Results for this configuration
Back-to-Back Testing using GridFTP
- memory to memory, 1 10GE NIC: 9.9 Gbps
- memory to memory, 4 10GE NICs: 39.5 Gbps
- disk to disk: 9.2 Gbps (1.2 GBytes/sec) using a single large file
Information on building a 100G DTN is here.