UDP Tuning
UDP will not get a full 10Gbps (or more) without some application level tuning as well. The important factors are:
- use jumbo frames if possible: performance will often be 4-5 times better using 9K MTUs
- packet size: best performance is MTU size minus packet header size. For example for a 9000Byte MTU, 8900 is safe value to try.
- socket buffer size: For UDP, buffer size is not related to RTT the way TCP is, but the defaults are still not large enough. Setting the socket buffer to 4M seems to help a lot in most cases
- core selection: UDP at 10G is typically CPU limited, so it is important to pick the right core.
- set the sending rate to a value that will not overrun the receive host. For iperf3, you can do user-level pacing with -b. Typically around 10G is a reasonable target speed per flow.
Sample commands for iperf, iperf3, and nuttcp:
iperf3 -l8900 -T30 -u -w4m -b10G -A 4,4 -c remotehost
nuttcp -l8900 -T30 -u -w4m -R10G -i1 -xc4/4 remotehost
numactl -C 4 iperf -l8900 -T30 -u -w4m -b10G -c remotehost
You may need to try different cores to find the best one for your host. You can use 'mpstat -P ALL 1' to figure out which core is being used for NIC interrupt handling, and then try a core in the same socket, but not the same core. Note that 'top' is not reliable for this.
In general nuttcp seems to be the fastest for UDP. Note that you'll need nuttcp V7.1 or higher to get the "-xc" option.
Even with this tuning, you'll need fast cores to get a full 10Gbps. For example, a 2.9GHz Intel Xeon CPU can get the full 10Gbps line rate, but with a 2.5GHz Intel Xeon CPU, we see only 5.9Gbps. The 2.9GHz CPU gets 22 Gbps of UDP using a 40G NIC.
If you want to see how much you can get with multiple UDP flows, each on a separate core, you can do something like this:
iperf3 -l8900 -T30 -T stream1 -u -w4m -b10G -A 4,4 -c remotehost & \
iperf3 -l8900 -T30 -T stream2 -u -w4m -b10G -A 5,5 -c remotehost
For more information on monitoring core usage, see our CPU Tuning page.

