Test/Measurement Host Tuning

Here is a quick reference guide for tuning settings for Linux Test/Measurement hosts such as perfSONAR hosts that run tools such as iperf, iperf3 or nuttcp.  Note that we are assuming the measurement tools are attempting to perform serialized single stream TCP tests with the perfSONAR pscheduler tool.  This assumption means we have increased the base TCP socket sizes to support larger memory allocations, and that parallel stream use is not as common as it would be for a tool such as GridFTP. For help in learning the appropriate size of the socket, you can use the BDP calculator. 

To support 10Gbps speeds, on paths of 100ms, 120MB of buffer must be available. You may want to adjust these values depending on the latency of the paths you want to test. Hosts configured to use jumbo frames need more buffer space as well.

General Settings

# increase TCP max buffer size setable using setsockopt()
# allow testing with 256MB buffers
net.core.rmem_max = 268435456 
net.core.wmem_max = 268435456 
# increase Linux autotuning TCP buffer limits 
# min, default, and max number of bytes to use
# allow auto-tuning up to 128MB buffers
net.ipv4.tcp_rmem = 4096 87380 134217728
net.ipv4.tcp_wmem = 4096 65536 134217728
# don't cache ssthresh from previous connection
net.ipv4.tcp_no_metrics_save = 1
# recommended default congestion control is htcp
net.ipv4.tcp_congestion_control = htcp
# If you are using Jumbo Frames, also set this
net.ipv4.tcp_mtu_probing = 1
# recommended to enable 'fair queueing'
net.core.default_qdisc = fq

For hosts with 10G NIC optimized for network paths up to 200ms RTT, or a 40G NIC up on paths up to 50ms RTT, use these values instead of the settings above:

# increase TCP max buffer size setable using setsockopt()
net.core.rmem_max = 536870912 
net.core.wmem_max = 536870912 
# increase Linux autotuning TCP buffer limit
net.ipv4.tcp_rmem = 4096 87380 268435456
net.ipv4.tcp_wmem = 4096 65536 268435456

We also strongly recommend reducing the maximum flow rate to avoid bursts of packets that could overflow switch and receive host buffers.

For example for a 10G host, add this to a boot script:

    /sbin/tc qdisc add dev ethN root fq maxrate 9gbit