fasterdata.es.netfasterdata.es.netESnet Network Performance Knowledge Base

NIC Tuning

Intel e1000

If you're using e1000 chips (Intel 1GE, often integrated into motherboards) the driver defaults to 256 Rx and 256 Tx descriptors. This is because early versions of the chipset only supported this. All recent versions have 4096, but the driver doesn't autodetect this. Increasing the number of descriptors can improve performance dramatically on some hosts.

To check if you are using an e1000 driver, do this:

  ethtool -i eth0

On Linux, add this to /etc/modprobe.conf:

  alias eth0 e1000
options e1000 RxDescriptors=4096,4096 TxDescriptors=4096,4096

and then reboot.

You can also add this to /etc/rc.local to get the same result.

  ethtool -G rx 4096 tx 4096

To verify that this worked, run (as root): ethtool -g eth0

On FreeBSD, add this to /boot/loader.conf:

hw.em.rxd=4096
hw.em.txd=4096

and then reboot. To verify that this worked, run (as root):

sysctl dev.em.0.debug=1 

and look at /var/log/messages

Mellanox 10Gig NIC Tuning Tips for Linux

See the Mellanox Performance Tuning Guide.

Also see the Mellanox ConnectX-3 Tuning page.

Myricom 10Gig NIC Tuning Tips for Linux

The Myricom NIC provides a number of tuning knobs. In particular setting interrupt coalescing can to help throughput a great deal:

    /usr/sbin/ethtool -C ethN rx-usecs 75

For more information

Recommended: README.myri10ge-linux
  Myri10GE FAQ 
  Myricom Performance Tuning Guide
Myri10GE ethtool output documentation

In particular, we've seen very dramatic improvements in firewalled environments using the Mryicom "Throttle" option. We have also seen up to 25% improvement using the latest driver downloaded from myricom and complied from source instead of the default driver from the Redhat/CentOS release.

Chelsio 10Gig NIC, Linux and FreeBSD

It has been reported that both TCP Segmentation Offloading (TSO) and TCP Offload Engine (TOE) on the Chelsio NIC radically hurt performance on a WAN (they do help reduce CPU load without affecting throughput on a LAN).

To turn off TSO do this:

 ethtool -K interface tso off

To disable TOE requires using the Chelsio "nic" driver instead of the "toe" driver. See the README file that comes with the driver source code available at http://service.chelsio.com/ for more information.

Chelsio also provides a script called 'perftune.sh' to tune a variety of settings on Linux.