Data Transfer Node Reference Implementation

ESnet has assembled a reference implementation of a host that can be deployed as a DTN or as a high-speed GridFTP test machine. The host can easily fill a 10Gbps network connection with disk-to-disk data transfers using GridFTP, and has demonstrated memory to memory transfers of 39.5 Gbps TCP using a 40G NIC.

The total cost of this server was around $12K in 2014, which included 16 250G SATA-3 HDD drives, which can sustain 2.2GByte/sec (17.7Gbps) reading from disk . Depending on your DTN capacity and disk IO requirements, the cost could be considerably more. Our server is 2U but you might want to consider getting a 4U to give you more room for future expansion.

Configuration Details

Hardware description

  • Motherboard: SuperMicro X9DRi-F
  • CPU: 2 x Intel(R) Xeon Ivy Bridge E5-2643V2 3.5GHz 6 Cores (Total 12 Cores)
  • Memory: 96G ( (12) 8GB DDR3-1866MHz RAM ECC/REG  )
  • RAID: Adaptec ASR-81605ZQ (16 ports)
  • 16 x Western Digital, model WD2500BHTZ-0
  • Network Controller:
    • 10G: Intel X520, Myricom 10G-PCIE2-8C2-2S, Chelsio T5
    • 40G: Mellanox MCX312A-XCBT
    • 100G:  Mellanox MCX455A-ECAT

System Configuration

We use the most recent CentOS-7 distribution of Linux, and have configured the data drives as RAID6.

Detailed Tuning Information is here.

Performance Results for this configuration

Back-to-Back Testing using GridFTP

  • memory to memory, 1 10GE NIC: 9.9 Gbps
  • memory to memory, 4 10GE NICs:  39.5 Gbps
  • disk to disk: 9.2 Gbps  (1.2 GBytes/sec) using a single large file

100G DTN

Information on building a 100G DTN is here.