Flow Control

Ethernet Flow Control

Flow control (sometimes called Priority Flow Control or PFC) allows a device receiving Ethernet frames to notify the sender that the receiver is having difficulty processing the frames it is being sent - this typically occurs when the receiver is temporarily overwhelmed and in danger of dropping packets.  The receiver can send a pause frame to request that the sender back off.  This is typically preferable to packet loss, especially when TCP-based bulk data transfer tools are the cause of the high packet rate.  If flow control is enabled on the host interface and on the interface of the switch or router, this packet loss can be avoided.

An example of this behavior is explored here.  The throughput test mesh on the ESnet perfSONAR dashboard showed that throughput tests involving a particular perfSONAR node were achieving less than expected performance.  Additional ad hoc testing appeared to indicate packet loss.  A look at the host indicated that the host itself was dropping packets (specifically, dropped_link_overflow was incrementing during tests, indicating that the host was receiving packets more quickly than it could process them):

[root@atla-pt1 ~]# ethtool -S eth1 | grep drop
     rx_dropped: 0
     tx_dropped: 0
     dropped_link_overflow: 8443
     dropped_link_error_or_filtered: 0
     dropped_pause: 0
     dropped_bad_phy: 0
     dropped_bad_crc32: 0
     dropped_unicast_filtered: 0
     dropped_multicast_filtered: 44835
     dropped_runt: 0
     dropped_overrun: 0
     dropped_no_small_buffer: 0
     dropped_no_big_buffer: 0

The host in question is a high-performance 10GE perfSONAR test machine connected to a Juniper MX960 router.  Examination of the router interface config indicated that the MX960 had flow control disabled:

interfaces {
    xe-1/3/0 {
        vlan-tagging;
        mtu 9192;
        encapsulation flexible-ethernet-services;
        gigether-options {
            no-flow-control;
        }
        ...
    }
}

Once the configuration that disabled flow control was removed, the packet loss was eliminated and performance increased from about 1.4Gbps to almost 8Gbps, as shown in this plot: