iperf2 / iperf3

iperf is a simple tool to let you measure memory-to-memory performance access a network. iperf3 is a newer implementation from scratch, with the goal of a smaller, simpler code base, and a library version of the functionality that can be used in other programs. iperf3 is not backwards compatible with iperf2.

The most recent iperf3 release is always available for both Debian and RHEL-based systems from the perfSONAR repository, or can be found on github. iperf3 is also in the standard linux repositories, but is usually a bit out of date.

We like iperf3 over older versions iperf2 because the TCP retransmit and CWND report is very helpful for troubleshooting (Note that iperf2.0.8 and higher now support this via the '-e' flag.), and the iperf3 option for JSON output makes results easy to parse. A comparison of throughput measurement tools is available here.

The basic commands are the same for iperf2 and iperf3:

iperf3 adds a number of additional features. For example, the -i mode now reports TCP retransmit info (and is on by default), and the verbose mode now gives a lot of useful information on CPU usage, etc. Other new options include:

iperf3 is normally used to measure memory to memory performance, but you can also use iperf3 for determine if the network or the disk is the bottleneck.

iperf3 thread model

In order to keep the code as simple and maintainable as possible, iperf3 was originally  single threaded. This meant that iperf3 would be CPU-bound on older hosts, or always on 40G/100G NICs. To run parallel stream iperf3 on multiple cores, you needed to use the method shown in the table above, or use iperf2, which is multi-treaded.

Starting with iperf3 v3.16, multiple threads are supported, eliminating this constraint.

For more information, see the iperf3 FAQ.

iperf2

Note that older versions of iperf were buggy. Most of the important bugs were fixed iperf2.0.8, and even newer versions have a number of nice features, so be sure to use a newer release of iperf2. 

iperf3 Omit Flag

The "omit" flag (-O) allows a specified number of seconds to be removed from a test result.  This was created to remove the ‘slow start’ portion of a TCP test, and focus on the steady state period that occurs.  For example, a normal TCP based iperf3 test  looks like the following:

$ iperf3 -c hostname

SENDER START
Connecting to host REMOTE, port 5207
[ 15] local LOCAL port 44605 connected to REMOTE port 5207
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[ 15]   0.00-1.00   sec  6.28 MBytes  52.6 Mbits/sec    0   1.32 MBytes       
[ 15]   1.00-2.00   sec   456 MBytes  3826 Mbits/sec    0   72.1 MBytes       
[ 15]   2.00-3.00   sec  1.12 GBytes  9658 Mbits/sec    0   72.1 MBytes       
[ 15]   3.00-4.00   sec  1.12 GBytes  9658 Mbits/sec    0   72.1 MBytes       
[ 15]   4.00-5.00   sec  1.12 GBytes  9657 Mbits/sec    0   72.1 MBytes       
[ 15]   5.00-6.00   sec  1.12 GBytes  9658 Mbits/sec    0   72.1 MBytes       
[ 15]   6.00-7.00   sec  1.12 GBytes  9655 Mbits/sec    0   72.1 MBytes       
[ 15]   7.00-8.00   sec  1.12 GBytes  9659 Mbits/sec    0   72.1 MBytes       
[ 15]   8.00-9.00   sec  1.12 GBytes  9658 Mbits/sec    0   72.1 MBytes       
[ 15]   9.00-10.00  sec  1.12 GBytes  9657 Mbits/sec    0   72.1 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[ 15]   0.00-10.00  sec  9.45 GBytes  8114 Mbits/sec    0             sender
[ 15]   0.00-10.00  sec  9.43 GBytes  8103 Mbits/sec                  receiver

iperf Done.

SENDER END

The average will be reported as 8.1 Gbps, even though the majority of the time was spent at 9.6 Gbps. Using the Omit flag, we see the following happen in the data, and final reported average:

$ iperf3 -O 3 -c hostname 

SENDER START
Connecting to host REMOTE, port 5208
[ 15] local LOCAL port 37535 connected to REMOTE port 5208
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[ 15]   0.00-1.00   sec  4.96 MBytes  41.6 Mbits/sec    0    987 KBytes       (omitted)
[ 15]   1.00-2.00   sec   332 MBytes  2786 Mbits/sec    0   65.4 MBytes       (omitted)
[ 15]   2.00-3.00   sec  1.12 GBytes  9636 Mbits/sec    0   72.1 MBytes       (omitted)
[ 15]   0.00-1.00   sec  1.12 GBytes  9657 Mbits/sec    0   72.1 MBytes       
[ 15]   1.00-2.00   sec  1.12 GBytes  9657 Mbits/sec    0   72.1 MBytes       
[ 15]   2.00-3.00   sec  1.12 GBytes  9657 Mbits/sec    0   72.1 MBytes       
[ 15]   3.00-4.00   sec  1.12 GBytes  9657 Mbits/sec    0   72.1 MBytes       
[ 15]   4.00-5.00   sec  1.12 GBytes  9657 Mbits/sec    0   72.1 MBytes       
[ 15]   5.00-6.00   sec  1.12 GBytes  9657 Mbits/sec    0   72.1 MBytes       
[ 15]   6.00-7.00   sec  1.13 GBytes  9666 Mbits/sec    0   72.1 MBytes       
[ 15]   7.00-8.00   sec  1.12 GBytes  9658 Mbits/sec    0   72.1 MBytes       
[ 15]   8.00-9.00   sec  1.12 GBytes  9657 Mbits/sec    0   72.1 MBytes       
[ 15]   9.00-10.00  sec  1.12 GBytes  9660 Mbits/sec    0   72.1 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[ 15]   0.00-10.00  sec  11.2 GBytes  9659 Mbits/sec    0             sender
[ 15]   0.00-10.00  sec  11.3 GBytes  9719 Mbits/sec                  receiver

iperf Done.

SENDER END