When sending from a faster host to a slower host, it is easy to overrun the receiver, leading to packet loss and TCP backing off. Similar problems occur when a 10G host sends data to a sub-10G virtual circuit, or a 40G host sending to a 10G host, or a 40G/100G host with a fast CPU sender to a 40G/100G host with a slower CPU. These issues are even more pronounced when using tools that use parallel streams, such as GridFTP. On some long paths (50-80ms RTT), we've seen TCP performance improvements of 2-4x after enabling packet pacing.
Fair Queuing (FQ)-based pacing, described below, is a very effective way of dealing with this issue. FQ pacing is relatively low-overhead compared to the older methods of pacing described here.
A presentation with several plots showing the advantages of FQ is available here.
Packet Pacing using the FQ (Fair Queuing) scheduler
Starting with the Linux kernel 3.11 or higher (available in Fedora 20, Debian 8, and Ubuntu 13.10), there is a new 'fair queuing' scheduler, which includes code that does a much better job of pacing packets out of a fast host. See https://lwn.net/Articles/564978/ for more details. For RHEL-based OSes, FQ has been backported to the 3.10.0-327 kernel in v7.2.
To enable Fair Queuing (which is off by default), do:
tc qdisc add dev $ETH root fq
or you can also add this to /etc/sysctl.conf:
net.core.default_qdisc = fq
To both pace and shape the bandwidth:
tc qdisc add dev $ETH root fq maxrate Ngbit
We recommending turning on FQ on all hosts connected to fast networks. For a 10G data transfer node (DTN) running GridFTP, which uses 4 parallel streams be default, we recommend setting FQ as follows:
tc qdisc add dev $ETH root fq maxrate 2.5gbit
You can also add FQ-based pacing to your application using the 'setsockopt' system call with the SO_MAX_PACING_RATE option. This only works if the host is configured to use fq as the qdisc however. The iperf3 tool uses this for the --fq-rate option (v3.1.5 and higher).
Note that TSO (which is off by default on most NICs) should be disabled when using FQ.
Also note that fq should not be confused with fq-codel, which is primarily for use on home networks connected via cable modems or DSL lines.