Test/Measurement Host Tuning

January 15, 2025

Here is a quick reference guide for tuning settings for Linux Test/Measurement hosts such as perfSONAR hosts that run tools such as iperf, iperf3 or nuttcp. Note that we are assuming the measurement tools are attempting to perform serialized single stream TCP tests with the perfSONAR pscheduler tool. This assumption means we have increased the base TCP socket sizes to support larger memory allocations, and that parallel stream use is not as common as it would be for a tool such as Globus. For help in learning the appropriate size of the socket, you can use the BDP calculator.

To support 10Gbps speeds, on paths of 100ms, 120MB of buffer must be available. You may want to adjust these values depending on the latency of the paths you want to test. Hosts configured to use jumbo frames need more buffer space as well.

For 100G measurement hosts, also see our 100G Network Tuning Guide.

General Settings

# increase TCP max buffer size setable using setsockopt()
# allow testing with 256MB buffers
net.core.rmem_max = 268435456 
net.core.wmem_max = 268435456 
# increase Linux autotuning TCP buffer limits 
# min, default, and max number of bytes to use
# allow auto-tuning up to 128MB buffers
net.ipv4.tcp_rmem = 4096 87380 134217728
net.ipv4.tcp_wmem = 4096 65536 134217728
# don't cache TCP metrics from previous connection
net.ipv4.tcp_no_metrics_save = 1
# If you are using Jumbo Frames, also set this
net.ipv4.tcp_mtu_probing = 1
# recommended to enable 'fair queueing' (fq or fq_codel)
net.core.default_qdisc = fq

For hosts with 10G NIC optimized for network paths up to 200ms RTT, or a 40G NIC up on paths up to 50ms RTT, use these values instead of the settings above:

# increase TCP max buffer size setable using setsockopt() to 512MB
net.core.rmem_max = 536870912 
net.core.wmem_max = 536870912 
# increase Linux autotuning TCP buffer limit to 256MB
net.ipv4.tcp_rmem = 4096 87380 268435456
net.ipv4.tcp_wmem = 4096 65536 268435456

For a host with a 100G NIC sending data across network paths up to 200ms RTT, you'll probably want to set the buffers to use the maximum value of 2GB.

# Allow buffers up to 2GB
net.core.rmem_max=2147483647 
net.core.wmem_max=2147483647
# increase Linux autotuning TCP buffer limit to 1GB
net.ipv4.tcp_rmem=4096 65536 1073741824
net.ipv4.tcp_wmem=4096 65536 1073741824

To apply your changes /etc/sysctl.conf, run:

sysctl -p

We also strongly recommend reducing the maximum flow rate to avoid bursts of packets that could overflow switch and receive host buffers.

For example, for a host with a 10G NIC or faster, add this to a boot script:

    /sbin/tc qdisc add dev ethN root fq maxrate 8gbit