Test/Measurement Host Tuning

Here is a quick reference guide for tuning settings for Linux Test/Measurement hosts such as perfSONAR hosts that run tools such as bwctl, iperf or nuttcp.  Note that we are assuming the measurement tools are attempting to perform serialized single stream TCP tests with the BWCTL tool.  This assumption means we have increased the base TCP socket sizes to support larger memory allocations, and that parallel stream use is not as common as it would be for a tool such as GridFTP. For help in learning the approproate size of the socket, you can use the BDP calculator

To support 10Gbps speeds, on paths of 100ms, 120MB of buffer must be available. You may want to adjust these values depending on the latency of the paths you want to test. Hosts configured to use jumbo frames need more buffer space as well.

General Settings

# increase TCP max buffer size setable using setsockopt()
# allow testing with 256MB buffers
net.core.rmem_max = 268435456
net.core.wmem_max = 268435456
# increase Linux autotuning TCP buffer limits
# min, default, and max number of bytes to use
# allow auto-tuning up to 128MB buffers
net.ipv4.tcp_rmem = 4096 87380 134217728
net.ipv4.tcp_wmem = 4096 65536 134217728
# recommended to increase this for CentOS6 with 10G NICS or higher
net.core.netdev_max_backlog = 250000
# don't cache ssthresh from previous connection
net.ipv4.tcp_no_metrics_save = 1
# Explicitly set htcp as the congestion control: cubic buggy in older 2.6 kernels
net.ipv4.tcp_congestion_control = htcp
# If you are using Jumbo Frames, also set this
net.ipv4.tcp_mtu_probing = 1
# recommended for CentOS7/Debian8 hosts
net.core.default_qdisc = fq

For hosts with 10G NIC optimized for network paths up to 200ms RTT, or a 40G NIC up on paths up to 50ms RTT, use these values instead of the settings above:

# increase TCP max buffer size setable using setsockopt()
net.core.rmem_max = 536870912
net.core.wmem_max = 536870912
# increase Linux autotuning TCP buffer limit
net.ipv4.tcp_rmem = 4096 87380 268435456
net.ipv4.tcp_wmem = 4096 65536 268435456
Add to /etc/rc.local:
# increase txqueuelen for CentOS6 hosts with 10G NICS
/sbin/ifconfig eth2 txqueuelen 10000