MTU Issues

Jumbo Ethernet frames can increase performance by up to factor of 2 on 10G paths on older hardware, particularly for UDP. For 40G paths, the performance gains are even greater.  Single stream 40G tests of TCP were observed to be 25Gbps vs. 10Gbps.  UDP testing showed an improvement of 15Gbps vs. 8Gbps. 

Use of Jumbo frames has two advantages:  One is that for a given data throughput, the packet rate is less for jumbo frames vs. standard frames, requiring less CPU for packet processing.  The other advantage is that recovery in data throughput after a loss event is proportional to the maximum segment size - so with jumbo frames you get a ~6x faster recovery rate from loss events.

ping can be used to verify the MTU size. For example, on Linux you can do:

 ping -s 8972 -M do -c 4 10.200.200.12

Scamper or tracepath are good tools to help verify the path MTU size too.

Note that there are some downsides to jumbo frames as well. All hosts in a single broadcast domain must to be configured with the same MTU, and this can be difficult and error-prone.  Ethernet has no way of detecting an MTU mismatch - this is a layer 3 function that requires ICMP signaling in order to work correctly. (Unfortunately some sites block ICMP, which breaks path MTU discovery. If tracepath fails, that is likely what is happening).

Therefore a good approach is often to create a new jumbo frame enabled subnet for your high-speed data transfer hosts.

For Linux hosts using Jumbo Frames, we recommend setting tcp_mtu_probing = 1 to help avoid the problem of MTU black holes. Setting it to 2 sometimes causes performance problems. For more information on this setting, see http://kb.pert.geant.net/PERTKB/PathMTU  and https://www.fir3net.com/Networking/Protocols/path-mtu-path-mtu-black-holes.html