If you're using e1000 chips (Intel 1GE, often integrated into motherboards) the driver defaults to 256 Rx and 256 Tx descriptors. This is because early versions of the chipset only supported this. All recent versions have 4096, but the driver doesn't autodetect this. Increasing the number of descriptors can improve performance dramatically on some hosts.
To check if you are using an e1000 driver, do this:
ethtool -i eth0
On Linux, add this to /etc/modprobe.conf:
alias eth0 e1000
options e1000 RxDescriptors=4096,4096 TxDescriptors=4096,4096
and then reboot.
You can also add this to /etc/rc.local to get the same result.
ethtool -G ethN rx 4096 tx 4096
To verify that this worked, run (as root): ethtool -g eth0
On FreeBSD, add this to /boot/loader.conf:
and then reboot. To verify that this worked, run (as root):
and look at /var/log/messages
Mellanox 10Gig NIC Tuning Tips for Linux
See the Mellanox Performance Tuning Guide.
Also see the Mellanox ConnectX-3 Tuning page.
Myricom 10Gig NIC Tuning Tips for Linux
The Myricom NIC provides a number of tuning knobs. In particular setting interrupt coalescing can to help throughput a great deal:
/usr/sbin/ethtool -C ethN rx-usecs 75
For more information
|Myricom Performance Tuning Guide|
|Myri10GE ethtool output documentation|
In particular, we've seen very dramatic improvements in firewalled environments using the Mryicom "Throttle" option. We have also seen up to 25% improvement using the latest driver downloaded from myricom and complied from source instead of the default driver from the Redhat/CentOS release.
Chelsio 10Gig NIC, Linux and FreeBSD
It has been reported that both TCP Segmentation Offloading (TSO) and TCP Offload Engine (TOE) on the Chelsio NIC radically hurt performance on a WAN (they do help reduce CPU load without affecting throughput on a LAN).
To turn off TSO do this:
ethtool -K interface tso off
To disable TOE requires using the Chelsio "nic" driver instead of the "toe" driver. See the README file that comes with the driver source code available at http://service.chelsio.com/ for more information.
Chelsio also provides a script called 'perftune.sh' to tune a variety of settings on Linux.