Interface Speed Mismatch Issues

One should be aware that for some network paths, sending from fast host to a slow host can cause problems. For example, a 10G host to a 1G host, or a 40G host to a 10G host. Here is an example showing tests between a 10G host and a 1G host:

A network interface card is designed to operate in two modes:

• If data is present, sent at line rate
• If data is not present, send nothing

This binary operation means that a smooth "average" speed is never possible, and as such devices in the path must be ready to accept the line rate burst of data, or risk dropping some due to a lack of buffer space. The speed mismatch happens when there is a lack of available network buffering to handle the influx of data and drop packets, as shown by the blue line above. The "slower" host sending to the "faster" host does not see this issue, and performance is fine. Using the tools tcpdump, tcptrace, and XPlot, we can see the reasons for this performance difference by observing the behavior of TCP:

The bursts of red indicate TCP retransmissions, and the purple is TCP SACK blocks.  This stalling leads to the often unpredictable graphs that show a low average throughput. 

This situation has been shown to be true in several environments:

• 10G to 1G host
• 10G host to a 2G virtual circuit
• 40G to 10G host
• Fast host to slower host (based on CPU speed)

Note that not all 10G to 1G mismatches will produce this behavior.  If there is sufficient buffering in the network (e.g. enough to handle the 10Gbps bursts that the faster network card may produce), the problem will not exist.  Consider this example for a 10G to 1G test on a network path with sufficient buffering:

The tcpdump for this path shows a typical TCP linear growth pattern - indicative of no stalls during the length of the transfer due to packet loss:

Many applications feature the ability to "pace" their traffic (e.g. nuttcp and iperf3 allow this behavior).  While this works well for UDP, it is hard to accurately pace TCP due to the way the kernel does buffering. The linux tc command does a much better job of pacing TCP flows, but only seems to be work on to speeds of around 9Gbps, even with a 40G NIC.

A full write up on tc can be found at https://fasterdata.es.net/performance-testing/packet-pacing/.  A final graph shows tcpdump results when using tc  to pace the sender speed to the receive host speed, and deliver much better performance:

Newer releases of the Linux Kernel are offering better control over speed mismatch.  The following graph of results shows sending from a 40G host to a 10G host, using both Linux Kernel 3.10 (CentOS 7) and Linux Kernel 4.2 across a ~50ms path: