Menu

iperf / iperf3

iperf is a simple tool to let you measure memory-to-memory performance access a network. iperf3 is a new implementation from scratch, with the goal of a smaller, simpler code base, and a library version of the functionality that can be used in other programs. iperf3 is not backwards compatible with iperf2.x. The most recent iperf3 release is always available for both Debian and RHEL-based systems from the perfSONAR repository. We recommend iperf3 over iperf becuase the TCP retransmit and CWND report if very helpful for troubleshooting.

On public test hosts, we recommed that bwctl should be used to ensure only 1 iperf test runs at a time.

The basic commands are the same for iperf and iperf3:

Sample iperf/iperf3 commands
Server:
iperf/iperf3 -s Start server on the default port
iperf -s -w 32M -D / iperf3 -s -D
Start server with larger TCP window, and in daemon mode
iperf -i1 -u -s -p 5003 / iperf3 -s -p 5003
Start UDP server on port 5003, and give 1 sec interval reports
Client
iperf/iperf3 -c remotehost -i 1 -t 30 Run a 30 second tests, giving results every 1 second
iperff/iperf3 -c remotehost -i 1 -t 20 -r Run a test from remotehost to localhost
iperf/iperf3 -c remotehost -i 1 -t 20 -w 32M -P 4 Run a test with 4 parallel streams, and with a 32M TCP buffer
iperf/iperf3 -c remotehost -u -i 1 -b 200M
Run a 200 Mbps UDP test
 

iperf3 adds a number of additional features. For example, the -i mode now reports TCP retransmit info (and is on by default), and the verbose mode now gives a lot of useful information on CPU usage, etc. Other new options include:

New iperf3 commands
Client:
iperf3 -c remotehost -i.5 -0 2
Run the test for 2 seconds before collecting results, to allow for TCP slowstart to finish. (Omit mode)
iperf3 -Z -c remotehost Use the sendfile() system call for "Zero Copy" mode. This uses much less CPU.
iperf3 -c 192.168.12.12 -T s1 & iperf3 -c 192.168.12.13 -T s2 Run tests to multiple interfaces at once, and label the lines to indicate which test is which
iperf3 -c remotehost -J Output the results in JSON format for easy parsing.
iperf3 -A 4,4 -c remotehost Set the CPU affinity for the sender,receiver (cores are numbered from 0). This has the same affect as doing 'numactl -C 4 iperf3'.

iperf3 -c 10.20.1.20 -A2,2 -T "1" & ; iperf3 -c 10.20.1.20 -p 5400 -A3,3 -T "2" &

Run 2 streams on 2 different cores, and label each using the "-T" flag.

iperf3 thread model

In order to keep the code as simple and maintainable as possible, iperf3 is single threaded. This means that that you may be CPU-bound on some hosts, or on 40G/100G NICs. To run parallel stream iperf3 on mutiple cores, use the method shown in the table above.

Omit Flag

The "omit" flag (-o) allows a specified number of seconds to be removed from a test result.  This was created to remove the ‘slow start’ portion of a TCP test, and focus on the steady state period that occurs.  For example, a normal TCP based Iperf3 test via BWCTL looks like the following:

[zurawski@newy-pt1 ~]$ bwctl -T iperf3 -f m -t 10 -i 1 -c ps.ncar.xsede.org
bwctl: Using tool: iperf3
bwctl: 27 seconds until test results available

SENDER START
Connecting to host 128.117.212.249, port 5207
[ 15] local 198.124.238.54 port 44605 connected to 128.117.212.249 port 5207
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[ 15]   0.00-1.00   sec  6.28 MBytes  52.6 Mbits/sec    0   1.32 MBytes       
[ 15]   1.00-2.00   sec   456 MBytes  3826 Mbits/sec    0   72.1 MBytes       
[ 15]   2.00-3.00   sec  1.12 GBytes  9658 Mbits/sec    0   72.1 MBytes       
[ 15]   3.00-4.00   sec  1.12 GBytes  9658 Mbits/sec    0   72.1 MBytes       
[ 15]   4.00-5.00   sec  1.12 GBytes  9657 Mbits/sec    0   72.1 MBytes       
[ 15]   5.00-6.00   sec  1.12 GBytes  9658 Mbits/sec    0   72.1 MBytes       
[ 15]   6.00-7.00   sec  1.12 GBytes  9655 Mbits/sec    0   72.1 MBytes       
[ 15]   7.00-8.00   sec  1.12 GBytes  9659 Mbits/sec    0   72.1 MBytes       
[ 15]   8.00-9.00   sec  1.12 GBytes  9658 Mbits/sec    0   72.1 MBytes       
[ 15]   9.00-10.00  sec  1.12 GBytes  9657 Mbits/sec    0   72.1 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[ 15]   0.00-10.00  sec  9.45 GBytes  8114 Mbits/sec    0             sender
[ 15]   0.00-10.00  sec  9.43 GBytes  8103 Mbits/sec                  receiver

iperf Done.

SENDER END

The average will be reported as 8.1 Gbps, even though the majority of the time was spent at 9.6 Gbps. Using the Omit flag, we see the following happen in the data, and final reported average:

[zurawski@newy-pt1 ~]$ bwctl -T iperf3 -f m -t 10 -i 1 -O 3 -c ps.ncar.xsede.org 
bwctl: Using tool: iperf3
bwctl: 30 seconds until test results available

SENDER START
Connecting to host 128.117.212.249, port 5208
[ 15] local 198.124.238.54 port 37535 connected to 128.117.212.249 port 5208
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[ 15]   0.00-1.00   sec  4.96 MBytes  41.6 Mbits/sec    0    987 KBytes       (omitted)
[ 15]   1.00-2.00   sec   332 MBytes  2786 Mbits/sec    0   65.4 MBytes       (omitted)
[ 15]   2.00-3.00   sec  1.12 GBytes  9636 Mbits/sec    0   72.1 MBytes       (omitted)
[ 15]   0.00-1.00   sec  1.12 GBytes  9657 Mbits/sec    0   72.1 MBytes       
[ 15]   1.00-2.00   sec  1.12 GBytes  9657 Mbits/sec    0   72.1 MBytes       
[ 15]   2.00-3.00   sec  1.12 GBytes  9657 Mbits/sec    0   72.1 MBytes       
[ 15]   3.00-4.00   sec  1.12 GBytes  9657 Mbits/sec    0   72.1 MBytes       
[ 15]   4.00-5.00   sec  1.12 GBytes  9657 Mbits/sec    0   72.1 MBytes       
[ 15]   5.00-6.00   sec  1.12 GBytes  9657 Mbits/sec    0   72.1 MBytes       
[ 15]   6.00-7.00   sec  1.13 GBytes  9666 Mbits/sec    0   72.1 MBytes       
[ 15]   7.00-8.00   sec  1.12 GBytes  9658 Mbits/sec    0   72.1 MBytes       
[ 15]   8.00-9.00   sec  1.12 GBytes  9657 Mbits/sec    0   72.1 MBytes       
[ 15]   9.00-10.00  sec  1.12 GBytes  9660 Mbits/sec    0   72.1 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[ 15]   0.00-10.00  sec  11.2 GBytes  9659 Mbits/sec    0             sender
[ 15]   0.00-10.00  sec  11.3 GBytes  9719 Mbits/sec                  receiver

iperf Done.

SENDER END

The database will store the 9.6 Gbps number, and feed this to graphing software.  Over time, this change will not inflate the test results, but rather compensate for problems seen during slow start of the test (e.g. if the window does not grow due to congestion, etc.). 

The ESnet dashboard (ps-dashboard.es.net) was modified to use this feature.  An example of the resulting graph appears below:

Note that impact on the data was not to raise throughput, but rather ‘normalize’ performance over time to make it appear more even. 

iperf3 is normally used to measure memory to memory performance, but you can also use iperf3 for determine if the network or the disk is the bottleneck.


Note that iperf 2.x results are inconsistent for UDP above 2Gbps. We recommend using nuttcp or iperf3 for high-speed UDP testing.