Intel 1GE and 10GE NICs
If you're using e1000 chips (Intel 1GE, often integrated into motherboards; note that this does not apply to newer variants) the driver defaults to 256 Rx and 256 Tx descriptors. This is because early versions of the chipset only supported this. All recent versions have 4096, but the driver doesn't autodetect this. Increasing the number of descriptors can improve performance dramatically on some hosts.
This is also the case with Intel 10GE NICs - the driver defaults to 256 descriptors each for Rx and Tx, even if the hardware supports 4096.
You can use ethtool to check on the number of descriptors your NIC has, and whether the driver is configured to use them. Some sample Linux output from an Intel 10GE NIC that is using the default config is below:
[user@perfsonar ~]# ethtool -g eth2
Ring parameters for eth2:
RX Mini: 0
RX Jumbo: 0
Current hardware settings:
RX Mini: 0
RX Jumbo: 0
Under Linux, to check which driver you are using, do this:
ethtool -i eth0
If you are using e1000 (1GE) or ixbge (10GE) and your hardware supports more descriptors than you are using, you can configure the driver to use the additional descriptors.
On Linux, add this to /etc/modprobe.conf (assuming e1000):
alias eth0 e1000
options e1000 RxDescriptors=4096,4096 TxDescriptors=4096,4096
and then reboot.
You can also add this to /etc/rc.local to get the same result.
ethtool -G ethN rx 4096 tx 4096
To verify that this worked, run (as root): ethtool -g eth0
On FreeBSD, add this to /boot/loader.conf:
and then reboot. To verify that this worked, run (as root):
and look at /var/log/messages
Mellanox 10Gig NIC Tuning Tips for Linux
See the Mellanox Performance Tuning Guide.
Also see the Mellanox ConnectX-3 Tuning page.
Myricom 10Gig NIC Tuning Tips for Linux
The Myricom NIC provides a number of tuning knobs. In particular setting interrupt coalescing can to help throughput a great deal:
/usr/sbin/ethtool -C ethN rx-usecs 75
For more information on the tradeoffs between rx/tx descriptors, interrupt coalescence, and L1 CPU cache size, see this article: http://patchwork.ozlabs.org/patch/348793/
For more information
|Myricom Performance Tuning Guide|
|Myri10GE ethtool output documentation|
In particular, we've seen very dramatic improvements in firewalled environments using the Mryicom "Throttle" option. We have also seen up to 25% improvement using the latest driver downloaded from myricom and complied from source instead of the default driver from the Redhat/CentOS release.
Centos 7 Issues
It has been reported that CentOS 7.1 does not enable Myricom's MSI-X by default, which decreases performance. To fix this, to the following:
Create a file /etc/modprobe.d/myri10ge.conf containing:
options myri10ge myri10ge_fw_name=myri10ge_rss_eth_z8e.dat myri10ge_gro=0 myri10ge_max_slices=-1
Chelsio 10Gig NIC, Linux and FreeBSD
It has been reported that both TCP Segmentation Offloading (TSO) and TCP Offload Engine (TOE) on the Chelsio NIC radically hurt performance on a WAN (they do help reduce CPU load without affecting throughput on a LAN). This is particularly true with a very fast CPU.
To turn off TSO do this:
ethtool -K interface tso off
To disable TOE requires using the Chelsio "nic" driver instead of the "toe" driver. See the README file that comes with the driver source code available at http://service.chelsio.com/ for more information.
Chelsio also provides a script called 'perftune.sh' to tune a variety of settings on Linux.