Menu

Setting Regular Testing Expectations

Regular Testing - Setting Expectation Levels

The following performance graph was captured between two hosts on the ESnet network, and represents "excellent" performance:

These hosts feature the following configurations:

  • Similar hardware age and capability
  • Similar operating system and TCP tunings
  • 60ms of Latency separation
  • Connected to ESnet devices, with plentiful capacity
  • A regularly configured TCP bandwidth test that runs the iperf tool for 20 seconds, and uses the "autotuning" feature of the Linux kernel (e.g. no set window size)

We called this "excellent" above, some may be asking how we can do this when it is approximately 50% of the available capacity for the host connections (10G).  To answer this fully, consider what happens when a 10 second test throughput test, similar to the one we run in the regular testing fashion, is run between two hosts.  

$ pscheduler task throughput --dest sunn-pt1.es.net

[ 16] local 198.129.254.58 port 5143 connected with 198.124.238.34 port 5143
[ ID] Interval       Transfer     Bandwidth
[ 16]  0.0- 2.0 sec  45.0 MBytes   189 Mbits/sec
[ 16]  2.0- 4.0 sec   834 MBytes  3499 Mbits/sec
[ 16]  4.0- 6.0 sec  1066 MBytes  4473 Mbits/sec
[ 16]  6.0- 8.0 sec  1065 MBytes  4468 Mbits/sec
[ 16]  8.0-10.0 sec  1065 MBytes  4466 Mbits/sec
[ 16]  0.0-10.0 sec  4076 MBytes  3415 Mbits/sec
[ 16] MSS size 8948 bytes (MTU 8988 bytes, unknown interface)

The emphasized part, the "summary" line at the end of the test, is what is recorded into the perfSONAR database.  Over the course of the test, with the limited amount of information we see, it is hard to say why we could not get to 9Gbps.  There are many plausible explanations:
  • TCP slow start takes several seconds to fully establish throughput, and the 20 second averaging the of the data will make things appear slower
  • Congestion could have forced TCP retransmissions, which would limit the growth of the autotuned window
  • The window settings on the host (set in /etc/sysctl.conf to maximize at 32MB) may not be sufficient for the BDP of the path
  • The single stream of TCP was not enough on its own to max out the available network capacity

Adjusting certain parameters, we can easily get to higher speeds:

  • Larger window sizes
    • $ pscheduler task throughput --window_size 128M --dest sunn-pt1.es.net 
      [ 16] local 198.129.254.58 port 5144 connected with 198.124.238.34 port 5144
      [ ID] Interval       Transfer     Bandwidth
      [ 16]  0.0- 2.0 sec  64.2 MBytes   269 Mbits/sec
      [ 16]  2.0- 4.0 sec  1119 MBytes  4694 Mbits/sec
      [ 16]  4.0- 6.0 sec  2192 MBytes  9193 Mbits/sec
      [ 16]  6.0- 8.0 sec  2192 MBytes  9195 Mbits/sec
      [ 16]  8.0-10.0 sec  2192 MBytes  9195 Mbits/sec
      [ 16]  0.0-10.0 sec  7762 MBytes  6510 Mbits/sec
      [ 16] MSS size 8948 bytes (MTU 8988 bytes, unknown interface)

  • Parallel streams
    • $ pscheduler task throughput --parallel 2 --dest sunn-pt1.es.net

      [ 17] local 198.124.238.34 port 56278 connected to 198.129.254.58 port 5855
      [ 20] local 198.124.238.34 port 35202 connected to 198.129.254.58 port 5855
      [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
      [ 17]   0.00-2.00   sec  40.0 MBytes   168 Mbits/sec    0   6.19 MBytes       
      [ 20]   0.00-2.00   sec  31.9 MBytes   134 Mbits/sec    0   4.27 MBytes       
      [SUM]   0.00-2.00   sec  71.9 MBytes   302 Mbits/sec    0             
      - - - - - - - - - - - - - - - - - - - - - - - - -
      [ 17]   2.00-4.00   sec   764 MBytes  3203 Mbits/sec    0   43.0 MBytes       
      [ 20]   2.00-4.00   sec   691 MBytes  2899 Mbits/sec    0   42.7 MBytes       
      [SUM]   2.00-4.00   sec  1.42 GBytes  6102 Mbits/sec    0             
      - - - - - - - - - - - - - - - - - - - - - - - - -
      [ 17]   4.00-6.00   sec  1016 MBytes  4263 Mbits/sec    0   43.0 MBytes       
      [ 20]   4.00-6.00   sec  1009 MBytes  4231 Mbits/sec    0   42.7 MBytes       
      [SUM]   4.00-6.00   sec  1.98 GBytes  8494 Mbits/sec    0             
      - - - - - - - - - - - - - - - - - - - - - - - - -
      [ 17]   6.00-8.00   sec  1012 MBytes  4247 Mbits/sec    0   43.0 MBytes       
      [ 20]   6.00-8.00   sec  1012 MBytes  4247 Mbits/sec    0   42.7 MBytes       
      [SUM]   6.00-8.00   sec  1.98 GBytes  8494 Mbits/sec    0             
      - - - - - - - - - - - - - - - - - - - - - - - - -
      [ 17]   8.00-10.00  sec  1010 MBytes  4235 Mbits/sec    0   43.0 MBytes       
      [ 20]   8.00-10.00  sec  1019 MBytes  4272 Mbits/sec    0   42.7 MBytes       
      [SUM]   8.00-10.00  sec  1.98 GBytes  8507 Mbits/sec    0             
      - - - - - - - - - - - - - - - - - - - - - - - - -
      [ ID] Interval           Transfer     Bandwidth       Retr
      [ 17]   0.00-10.00  sec  3.75 GBytes  3223 Mbits/sec    0             sender
      [ 17]   0.00-10.00  sec  3.75 GBytes  3223 Mbits/sec                  receiver
      [ 20]   0.00-10.00  sec  3.67 GBytes  3157 Mbits/sec    0             sender
      [ 20]   0.00-10.00  sec  3.67 GBytes  3156 Mbits/sec                  receiver
      [SUM]   0.00-10.00  sec  7.43 GBytes  6380 Mbits/sec    0             sender
      [SUM]   0.00-10.00  sec  7.43 GBytes  6380 Mbits/sec                  receiver


  • Longer test duration
$ pscheduler task throughput --duration PT20S  --dest sunn-pt1.es.net
[ 16] local 198.129.254.58 port 5153 connected with 198.124.238.34 port 5153
[ ID] Interval       Transfer     Bandwidth
[ 16]  0.0- 5.0 sec  1932 MBytes  3241 Mbits/sec
[ 16]  5.0-10.0 sec  3735 MBytes  6266 Mbits/sec
[ 16] 10.0-15.0 sec  4961 MBytes  8323 Mbits/sec
[ 16] 15.0-20.0 sec  5279 MBytes  8857 Mbits/sec
[ 16]  0.0-20.0 sec  15907 MBytes  6669 Mbits/sec
[ 16] MSS size 8948 bytes (MTU 8988 bytes, unknown interface)

  • Or increasing the default tuning settings in /etc/sysctl.conf
    • net.ipv4.tcp_rmem = 4096 87380 134217728
      net.ipv4.tcp_wmem = 4096 65536 134217728

In general, poor performance is easy to spot.  Packet loss causes TCP to behave erratically on short RTT tests, and perform extremely poorly as the RTT increases.  The next sections will outline this.