MTU Issues

Jumbo Ethernet frames can increase performance by up to factor of 2 on 10G paths on older hardware, particularly for UDP. For 40/100G paths, the performance gains are even greater.  Single stream 100G TCP tests across ESnet are typically 3x faster with jumbo frames.   

Use of Jumbo frames has two advantages:  One is that for a given data throughput, the packet rate is less for jumbo frames vs. standard frames, requiring less CPU for packet processing.  The other advantage is that recovery in data throughput after a loss event is proportional to the maximum segment size - so with jumbo frames you get a ~6x faster recovery rate from loss events.

ping can be used to verify the MTU size. For example, on Linux you can do:

 ping -4 -s 8940 -M do -c 4 hostname  (for IPV4)

 ping -6 -s 8900 -M do -c 4 hostname  (for IPV6)

 Note that not all versions of ping support the '-M flag'.

Scamper and tracepath are good tools to help verify the path MTU size too.

Note that there are some downsides to jumbo frames as well. All hosts in a single broadcast domain must to be configured with the same MTU, and this can be difficult and error-prone.  Ethernet has no way of detecting an MTU mismatch - this is a layer 3 function that requires ICMP signaling in order to work correctly. (Unfortunately some sites block ICMP, which breaks path MTU discovery. If tracepath fails, that is likely what is happening).

Therefore a good approach is often to create a new jumbo frame enabled subnet for your high-speed data transfer hosts.

For Linux hosts using Jumbo Frames, we recommend setting tcp_mtu_probing = 1 to help avoid the problem of MTU black holes. Setting it to 2 sometimes causes performance problems. For more information on this setting, see https://www.fir3net.com/Networking/Protocols/path-mtu-path-mtu-black-holes.html