fasterdata.es.netfasterdata.es.netESnet Network Performance Knowledge Base
Host Tuning

Linux Tuning

This page contains a quick reference guide for Linux 2.6 tuning for hosts connected at speeds of 1Gbps or higher. Note that most of the tuning settings described here will actually decrease performance of hosts connected at rates of OC3 (155 Mbps) or less, such as home users on Cable/DSL connections.

For a detailed explanation of some of the advice on this page, see the Linux Tuning Expert page. Also see the tuning advice for test/measurement hosts page.

 General Approach

To check what setting your system is using, use 'sysctl name' (e.g.: 'sysctl net.ipv4.tcp_rmem'). To change a setting use 'sysctl -w'. To make the setting permanent add the setting to the file 'sysctl.conf'.

 TCP tuning

Like most modern OSes, Linux now does a good job of auto-tuning the TCP buffers, but the  default maximum Linux TCP buffer sizes are too small. The following settings are recommended:

# increase TCP max buffer size setable using setsockopt()
# 16 MB with a few parallel streams is recommended for most 10G paths
# 32 MB might be needed for some very long end-to-end 10G or 40G paths
net.core.rmem_max = 16777216 
net.core.wmem_max = 16777216 
# increase Linux autotuning TCP buffer limits 
# min, default, and max number of bytes to use
# (only change the 3rd value, and make it 16 MB or more)
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
# recommended to increase this for 10G NICS
net.core.netdev_max_backlog = 30000
# these should be the default, but just to be sure
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1

Note: you should leave net.tcp_mem alone. The defaults are fine.

Linux supports pluggable congestion control algorithms . To get a list of congestion control algorithms that are available in your kernel (kernal  2.6.20+), run:

sysctl net.ipv4.tcp_available_congestion_control

If cubic and/or htcp are not listed try the following, as most distributions include them as loadable kernel modules:

/sbin/modprobe tcp_htcp
/sbin/modprobe tcp_cubic

NOTE: There seem to be bugs in both bic and cubic for a number of versions of the Linux kernel up to version 2.6.33. We recommend using htcp with older kernels to be safe. To set the congestion control do:

 sysctl -w net.ipv4.tcp_congestion_control=htcp

 UDP Tuning

The TCP window size actually effects UDP as well on Linux. Be sure to use the setsockopt() call to increase SO_SNDBUF/SO_RCVBUF to around 4MB if you want to do UDP streams that are faster than 3-4 Gbps. Also, on a network with 9K MTUs the optimal UDP packet size is 8972 bytes.

 NIC Tuning

These can be added to /etc/rc.local to get run at boot time.

# increase txqueuelen for 10G NICS
/sbin/ifconfig eth2 txqueuelen 10000

Note that this might have adverse affects for a 10G host sending to a 1G host or slower.


 Virtual Machine Tuning

To get the best network performance on a Linux VM running on a native Linux host, set txqueuelen in host OS, and set all other tuning parameters in the guest OS.