Menu

Mellanox ConnectX-3

We recommend using the latest device driver from Mellanox rather than the one in your Linux distribution.

Note that the Mellanox device driver installation script automatically adds the following to your /etc/sysctl.conf file:

## MLXNET tuning parameters ##
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_sack = 0
net.ipv4.tcp_low_latency = 1
net.core.netdev_max_backlog = 250000
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.rmem_default = 16777216
net.core.wmem_default = 16777216
net.core.optmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
## END MLXNET ##

Be aware that this overrides any tuning you have in that file! While we agree with these settings for good for a LAN, for a WAN you will need much larger TCP buffers, and we recommend turning on SACK.  Use something like this instead:

## FASTERDATA 40G tuning parameters ##
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_low_latency = 0
net.core.netdev_max_backlog = 250000
net.core.rmem_max = 268435456
net.core.wmem_max = 268435456
net.ipv4.tcp_rmem = 4096 87380 134217728
net.ipv4.tcp_wmem = 4096 65536 134217728
## END FASTERDATA ##

If you have 40G/100G NICs and are using the the non-OFED driver, you might also want to try modifying the following settings in  /etc/modprobe.d/mlx4.conf, as described at: https://community.mellanox.com/docs/DOC-2819

options mlx4_en inline_thold=0
options mlx4_core log_num_mgm_entry_size=-7

Also, be sure that adaptive-rx is off:

ethtool -C eth0 adaptive-rx off