MTU Issues

Jumbo Ethernet frames can increase performance by a factor of 2-4 on 10G paths.

Jumbo frames have two advantages:  One is that for a given data throughput, the packet rate is less for jumbo frames vs. standard frames, requiring less CPU for packet processing.  The other advantage is that recovery in data throughput after a loss event is proportional to the maximum segment size - so with jumbo frames you get a ~6x faster recovery rate from loss events. For more information on the benefits of Jumbo Frames, see: http://staff.psc.edu/mathis/MTU/

ping can be used to verify the MTU size. For example, on Linux you can do:

 ping -s 8972 -M do -c 4 10.200.200.12

Scamper or tracepath are good tools to help verify the path MTU size too.

Note that all hosts in a single broadcast domain need to be configured with the same MTU.  Ethernet has no way of detecting an MTU mismatch - this is a layer 3 function that requires ICMP signaling in order to work correctly. (Unfortunately some sites block ICMP, which breaks path MTU discovery. If tracepath fails, that is likely what is happening).

Therefore a good approach is often to create a new jumbo frame enabled subnet for your high-speed data transfer hosts.

For Linux hosts using Jumbo Frames, we recommend setting tcp_mtu_probing = 1 to help avoid the problem of MTU black holes. Setting it to 2 sometimes causes performance problems. For more information on this setting, see http://kb.pert.geant.net/PERTKB/PathMTU