Menu

nuttcp

Recent versions of nuttcp (versions 7.1.1+) have some new features that can be quite useful. (ESnet is using this version.) Here is a summary of some of these features.

server: nuttcp -S
client: nuttcp -i1 server_hostname
0.8125 MB / 1.00 sec = 6.8150 Mbps 0 retrans
0.7500 MB / 1.00 sec = 6.2907 Mbps 0 retrans
0.7500 MB / 1.00 sec = 6.2907 Mbps 0 retrans
0.8125 MB / 1.00 sec = 6.8150 Mbps 0 retrans
0.8125 MB / 1.00 sec = 6.8150 Mbps 0 retrans
0.7500 MB / 1.00 sec = 6.2907 Mbps 24 retrans
0.6875 MB / 1.00 sec = 5.7665 Mbps 17 retrans
0.8125 MB / 1.00 sec = 6.8150 Mbps 11 retrans
1.0625 MB / 1.00 sec = 8.9119 Mbps 3 retrans

Test the reverse direction:

   nuttcp -i1 -r server_hostname
UDP Burst mode

    Starting with version 6.2.8, nuttcp includes a 'burst mode' for UDP, which is useful to find paths that are constrained by network devices with not enough buffering.

    e.g.: Send 300 Mbps of UDP in bursts of 20 packets for 5 seconds

 ./nuttcp -u -Ri300m/20 -i 1 -T5 nettest.lbl.gov
35.7607 MB / 1.00 sec = 299.9241 Mbps 0 / 36619 ~drop/pkt 0.00 ~%loss
35.7617 MB / 1.00 sec = 300.0018 Mbps 0 / 36620 ~drop/pkt 0.00 ~%loss
35.5645 MB / 1.00 sec = 298.3730 Mbps 2 / 36420 ~drop/pkt 0.00549 ~%loss
33.3262 MB / 1.00 sec = 279.5649 Mbps 2253 / 36379 ~drop/pkt 6.19 ~%loss
34.6924 MB / 1.00 sec = 291.0208 Mbps 1095 / 36620 ~drop/pkt 2.99 ~%loss

175.1836 MB / 5.00 sec = 293.7936 Mbps 99 %TX 11 %RX 3350 / 182738 drop/pkt 1.83 %loss

This amount of loss is tolerable. Now send a 300 Mbps in bursts of 50 packets:

./nuttcp -u -Ri300m/50 -i 1 -T5 nettest.lbl.gov
23.8232 MB / 1.00 sec = 199.8394 Mbps 12238 / 36633 ~drop/pkt 33.41 ~%loss
25.2227 MB / 1.00 sec = 211.5836 Mbps 10783 / 36611 ~drop/pkt 29.45 ~%loss
25.1846 MB / 1.00 sec = 211.2405 Mbps 10816 / 36605 ~drop/pkt 29.55 ~%loss
24.2969 MB / 1.00 sec = 203.8392 Mbps 11754 / 36634 ~drop/pkt 32.08 ~%loss
25.1484 MB / 1.00 sec = 210.8927 Mbps 10864 / 36616 ~drop/pkt 29.67 ~%loss

123.7090 MB / 5.00 sec = 207.5136 Mbps 99 %TX 12 %RX 56470 / 183148 drop/pkt 30.83 %loss

And the level of loss become excessive, showing that some device in the path needs bigger buffers. Note that bursts of > 50 packets is common with TCP over paths with a RTT > 20 ms.

On a 10G network with a 9K MTU, the following has worked to identify a path with buffers that are too small:

If this has no packet loss: 

   nuttcp -l8972 -T30 -u -w4m -Ri300m/100 -i1 server_hostname

And this has packet loss:

   nuttcp -l8972 -T30 -u -w4m -Ri300m/300 -i1 server_hostname

Then likely there is a device with buffers less than 32MB per port in the path.


 

nuttcp also supports 3rd party mode, similar to bwctl:

 nuttcp -P 5099/5099 -T30 -i2 sendhost receivehost

Testing Multiple NICS, using CPU affinity binding:

    nuttcp -i1 -xc 2/2 -Is1 -p 5500 10.26.0.132 & nuttcp -i1 -xc 3/3 -Is2 -p 5501 10.26.0.132
10G UDP testing

nuttcp is definitely the best tool for high-speed UDP testing. To get a full 10Gbps using UDP requires the right MTU size (9K), the right packet size (8972), and an larger buffer size. For example:

nuttcp -l8972 -T30 -u -w4m -Ru -i1 192.168.2.11