NIC Tuning

Intel 1GE and 10GE NICs

If you're using e1000 chips (Intel 1GE, often integrated into motherboards; note that this does not apply to newer variants) the driver defaults to 256 Rx and 256 Tx descriptors. This is because early versions of the chipset only supported this. All recent versions have 4096, but the driver doesn't autodetect this. Increasing the number of descriptors can improve performance dramatically on some hosts.

This is also the case with Intel 10GE NICs - the driver defaults to 256 descriptors each for Rx and Tx, even if the hardware supports 4096.

You can use ethtool to check on the number of descriptors your NIC has, and whether the driver is configured to use them.  Some sample Linux output from an Intel 10GE NIC that is using the default config is below:

[[email protected] ~]# ethtool -g eth2
Ring parameters for eth2:
Pre-set maximums:
RX:          4096
RX Mini:     0
RX Jumbo:    0
TX:          4096
Current hardware settings:
RX:          256
RX Mini:     0
RX Jumbo:    0
TX:          256

Under Linux, to check which driver you are using, do this:

  ethtool -i eth0

If you are using e1000 (1GE) or ixgbe (10GE) and your hardware supports more descriptors than you are using, you can configure the driver to use the additional descriptors.

On Linux, add this to /etc/modprobe.conf (assuming e1000):

  alias eth0 e1000
  options e1000 RxDescriptors=4096,4096 TxDescriptors=4096,4096

and then reboot.

You can also add this to /etc/rc.local to get the same result.

  ethtool -G ethN rx 4096 tx 4096

To verify that this worked, run (as root): ethtool -g eth0

On FreeBSD, add this to /boot/loader.conf:

hw.em.rxd=4096
hw.em.txd=4096

and then reboot. To verify that this worked, run (as root):

sysctl dev.em.0.debug=1 

and look at /var/log/messages

Mellanox 10Gig NIC Tuning Tips for Linux

See the Mellanox Performance Tuning Guide.

Also see the Mellanox ConnectX-3 Tuning page.

Myricom 10Gig NIC Tuning Tips for Linux

The Myricom NIC provides a number of tuning knobs. In particular setting interrupt coalescing can to help throughput a great deal:

    /usr/sbin/ethtool -C ethN rx-usecs 75

For more information on the tradeoffs between rx/tx descriptors, interrupt coalescence, and L1 CPU cache size, see this article: http://patchwork.ozlabs.org/patch/348793/

For more information

In particular, we've seen very dramatic improvements in firewalled environments using the Mryicom "Throttle" option. We have also seen up to 25% improvement using the latest driver downloaded from myricom and complied from source instead of the default driver from the Redhat/CentOS release.

Centos 7 Issues

It has been reported that  CentOS 7.1 does not enable Myricom's MSI-X by default, which decreases performance. To fix this, to the following:

Create a file /etc/modprobe.d/myri10ge.conf containing:

options myri10ge myri10ge_fw_name=myri10ge_rss_eth_z8e.dat myri10ge_gro=0 myri10ge_max_slices=-1

Chelsio 10Gig NIC, Linux and FreeBSD

It has been reported that both TCP Segmentation Offloading (TSO) and TCP Offload Engine (TOE) on the Chelsio NIC radically hurt performance on a WAN (they do help reduce CPU load without affecting throughput on a LAN). This is particularly true with a very fast CPU.

To turn off TSO do this:

 ethtool -K interface tso off

To disable TOE requires using the Chelsio "nic" driver instead of the "toe" driver. See the README file that comes with the driver source code available at http://service.chelsio.com/ for more information.

Chelsio also provides a script called 'perftune.sh' to tune a variety of settings on Linux.