nuttcp

nuttcp is another network test tool, similar to iperf2 and iperf3, that has a number of unique features. perfSONAR currently includes version 8.1.4 of nuttcp. nuttcp version 7.1.1+ has some  features that can be quite useful. 

Here is a summary of some of these features.

server: nuttcp -S
client: nuttcp -i1 server_hostname
   0.8125 MB /   1.00 sec =    6.8150 Mbps     0 retrans
   0.7500 MB /   1.00 sec =    6.2907 Mbps     0 retrans
   0.7500 MB /   1.00 sec =    6.2907 Mbps     0 retrans
   0.8125 MB /   1.00 sec =    6.8150 Mbps     0 retrans
   0.8125 MB /   1.00 sec =    6.8150 Mbps     0 retrans
   0.7500 MB /   1.00 sec =    6.2907 Mbps    24 retrans
   0.6875 MB /   1.00 sec =    5.7665 Mbps    17 retrans
   0.8125 MB /   1.00 sec =    6.8150 Mbps    11 retrans
   1.0625 MB /   1.00 sec =    8.9119 Mbps     3 retrans

Test the reverse direction:

   nuttcp -i1 -r server_hostname

UDP Burst mode

Starting with version 6.2.8, nuttcp includes a 'burst mode' for UDP, which is useful to find paths that are constrained by network devices with not enough buffering.

e.g.: Send 300 Mbps of UDP in bursts of 20 packets for 5 seconds

 ./nuttcp -u -Ri300m/20 -i 1 -T5 server_hostname
   35.7607 MB /   1.00 sec =  299.9241 Mbps     0 / 36619 ~drop/pkt  0.00 ~%loss
   35.7617 MB /   1.00 sec =  300.0018 Mbps     0 / 36620 ~drop/pkt  0.00 ~%loss
   35.5645 MB /   1.00 sec =  298.3730 Mbps     2 / 36420 ~drop/pkt 0.00549 ~%loss
   33.3262 MB /   1.00 sec =  279.5649 Mbps  2253 / 36379 ~drop/pkt  6.19 ~%loss
   34.6924 MB /   1.00 sec =  291.0208 Mbps  1095 / 36620 ~drop/pkt  2.99 ~%loss

  175.1836 MB /   5.00 sec =  293.7936 Mbps 99 %TX 11 %RX 3350 / 182738 drop/pkt 1.83 %loss

This amount of loss is tolerable. Now send a 300 Mbps in bursts of 50 packets:

./nuttcp -u -Ri300m/50 -i 1 -T5 server_hostname
   23.8232 MB /   1.00 sec =  199.8394 Mbps 12238 / 36633 ~drop/pkt 33.41 ~%loss
   25.2227 MB /   1.00 sec =  211.5836 Mbps 10783 / 36611 ~drop/pkt 29.45 ~%loss
   25.1846 MB /   1.00 sec =  211.2405 Mbps 10816 / 36605 ~drop/pkt 29.55 ~%loss
   24.2969 MB /   1.00 sec =  203.8392 Mbps 11754 / 36634 ~drop/pkt 32.08 ~%loss
   25.1484 MB /   1.00 sec =  210.8927 Mbps 10864 / 36616 ~drop/pkt 29.67 ~%loss

  123.7090 MB /   5.00 sec =  207.5136 Mbps 99 %TX 12 %RX 56470 / 183148 drop/pkt 30.83 %loss

And the level of loss become excessive, showing that some device in the path needs bigger buffers. Note that bursts of > 50 packets is common with TCP over paths with a RTT > 20 ms.

On a 10G network with a 9K MTU, the following has worked to identify a path with buffers that are too small:

If this has no packet loss: 

   nuttcp -l8972 -T30 -u -w4m -Ri300m/100 -i1 server_hostname

And this has packet loss:

   nuttcp -l8972 -T30 -u -w4m -Ri300m/300 -i1 server_hostname

Then likely there is a device with buffers less than 32MB per port in the path.

nuttcp also supports 3rd party mode, similar to pscheduler:

 nuttcp -P 5099/5099 -T30 -i2 sendhost receivehost

Testing Multiple NICS, using CPU affinity binding:

    nuttcp -i1 -xc 2/2 -Is1 -p 5500 10.26.0.132 & nuttcp -i1 -xc 3/3 -Is2 -p 5501 10.26.0.132

10G+ UDP testing

nuttcp is definitely the best tool for high-speed UDP testing. To get a full 10Gbps using UDP requires the right MTU size (9K), the right packet size (8972), and an larger buffer size. For example:

nuttcp -l8972 -T30 -u -w4m -Ru -i1 192.168.2.11