Motherboard and Chassis selection

If you are buying a DTN node today (2022 as of this writing) and want your server to have a 3-5 year lifespan, you should plan for a system that will eventually have a 100G+ NIC, even if you have no immediate plans for 100G at your site. This means you need a host that supports PCI Express gen 3 (aka PCIe gen3), or ideally PCIe gen4. Fortunately, PCIe gen4 is a common option in modern server hardware. Other considerations are memory speed, number of PCI slots, extra cooling, and an adequate and redundant power supply option.

While there are several important aspects of hardware selection, one thing stands out: CPU clock rate is very important for high performance hosts. In general, for a DTN it is better to have a higher CPU clock rate than a higher core count.

Intel’s Xeon Scalable CPU (14 and 10nm) and AMD EPYC gen 2 and gen 3 architecture provide these features, which benefit a DTN:

PCIe Gen4 support ( up to 32 GB/sec )
Turbo boost (up to ~4GHz for Intel and AMD CPUs)
Faster interconnects for communication between processors

One interesting feature of these new processors is that PCI bus slots are connected directly to a processor. In a multi-processor system, this means your process might end up on one processor, but your I/O interrupts on another processor. When this happens there will be a huge performance hit. For example, high-speed NICs rates can saturate the interconnect that connects multiple processors and complicates performance tuning. Because of this, we recommend going for highest number of cores you can get on a single CPU, and avoid dual-processor motherboards if you only need a reduced number of PCIe expansion slots.

If you want to optimize for a small number of fast (> 10Gbps) flows, this means you will need to manage IQR bindings by hand. But if you are optimizing for many 500Mbps-1Gbps flows, this will be less of an issue. This also means you may be better off doing 4x10GE instead of 1x40GE, as you will have more control mapping IRQs to processors.

Memory

We recommend at least 64GB of RAM for a DTN node. More is better.

PCI Slots

Be sure to get the right number of the right type of PCI slots for your needs. PCI slots are defined by:

Form factor: This is the length of the slots, referred as the number of PCI lanes it can support. A 16 lane controller is twice as long as a 8 lane controller.
Number of wired lanes: not all lanes of the slot may be wired. Some 8 lanes controller may only have 4 lanes wired.

PCIe 2.0 is 500 MB/sec per lane. A typical host supports 8 lane (x8) cards, or up to 4 GB/sec. A high-end host might have 16 lane (x16) slots, or up to 8 GB/sec. PCIe 3.0 doubles this bandwidth and PCIe 4.0 doubles this bandwidth again.

Make sure the motherboard you order has the right number of slots with the right number of lanes for you planned usage. For example:

10GE NICs require an 8 lane PCIe-2 slot
40G/100G NICs require an 8/16 lane PCIe-3 slot
200G NICs require a 16 lane PCIe-4 slot
Most RAID controllers require 8 lane PCIe-2 slot
Very high-end RAID controllers for SSD might require a 16 lane PCIe-2 slot or a 8 line PCIe-3 slot
A high-end PCIe SSD drive requires a 16 lane PCIe-3 slot