100G DTN

If you wish to build a 100G DTN, here are the important hardware considerations:

  1. 100G NICs require PCIe Gen3 x16. If you are assuming all transfers are 4 or more parallel streams, any CPU with a clock rate greater than 2.5GHz should be fast enough to push 25Gbps per flow with the standard tuning applied.
  2. 100 Gbps (or 12.5 GBytes/sec) of disk IO is challenging.

Based on reports from colleagues, to get this much disk I/O you'll need either:

  • 10 SSD NVME PCIe Gen 3 x4 drives (e.g.: Samsung 950 Pro with U.2 to M.2 2.5" adapters), or 8 high-end NVME PCIe Gen 3 x4 drives (e.g.: Intel DC P3700 with U.2, 2.5" version that has high endurance).
  • 24 SSD SATA drives (with two PCIe Gen 3 x8 RAID controllers, or one PCIe Gen 3 x16 controller)

Note that both of these configurations will require a special chassis that can hold that many drives, such as:

Sample drives that have been used for 100G DTNs include:

  • SSD NVME drives:
    • Samsung 950 Pro or Intel DC P3700
  • SSD SATA drives:
    • Samsung PM863 or SM863

Recommendations for CPU, memory, and RAID controller from our reference implementation still apply as well.

Note: you can build a server with one processor with memory instead of two, and avoid the NUMA performance issues.

Additional information on tuning for 40/100G hosts can be found here.

Nikhef 100G Power DTN

Tristan Suerink at Nikhef in the Netherlands has some good information describing a 100G DTN / Storage host based on the IBM Power architecture in 2018.