Menu

DTN Tuning

Tuning your DTN host is extremely important. We have seen overall IO throughput of a DTN more than double with proper tuning.

Tuning can be as much art as a science. Due to differences in hardware, it's hard to give concrete running advice. In general you should attempt to tune one thing at a time, and runs some benchmarks to see if it made a difference. Some sample benchmark commands are shown here.

Here are some tuning settings that we have found do make a difference. Note that you should always use the most recent version of the OS, as performance optimizations for new hardware are added to every release.

Additional information on tuning for 40/100G/200G hosts can be found here.

Network

Network tuning is the most important thing to pay attention to. Be sure to following the the advice in our Linux Tuning Guide. We also recommend configuring FQ-based packet pacing. If you are trying to get as much bandwidth as possible out of your DTN, you'll also want to do Interrupt Binding.

Packet Pacing

DTN testing by ESnet and several others has confirmed that you will reduce packet loss and maximize DTN throughput by using packet pacing to throttle network traffic to 80-90% of the NIC speed. For example for a 10G NIC running GridFTP doing 4 parallel streams, we strongly recommend using 'tc' to set the per-stream maximum to 2Gbps. e.g.:

/sbin/tc qdisc add dev ethN root fq maxrate 2gbit

 

BIOS

For PCI gen3 and gen4-based hosts, you should enable “turbo boost”, and disable hyperthreading and node interleaving. More information on BIOS tunings are described in these documents: AMD and Intel (2.1 BIOS Recommendations).

If enabling IOMMU is desired for virtualization, ensure that iommu=pt is configured as a kernel boot option to avoid mapping overheads on the native host.

I/O Scheduler

The default scheduler on some versions of Linux is the "fair" scheduler. For a DTN node, we recommend using the "deadline" scheduler instead. To enable deadline scheduling, add "elevator=deadline" to the end of the "kernel' line in your /boot/grub/grub.conf file, similar to this:

kernel /vmlinuz-2.6.35.7 ro root=/dev/VolGroup00/LogVol00 rhgb quiet elevator=deadline

File System

We recommend using the xfs file system in Linux for DTN nodes. 

XFS configuration and tuning

mkfs defaults are recommended. However, there are guidelines for matching RAID stripe unit and stripe width to optimize for performance.

RAID Controller

Different RAID controllers provide different tuning controls. Check the documentation for your controller and use the settings recommended to optimize for large file reading. You will usually want to disable any “smart” controller built-in options, as they are typically designed for different workflows.