Performance Expectations for a 100G Host

January 17, 2024

So you've installed your shiny new 100G perfSONAR server and/or 100G DTN, what speeds should you expect to see out of the box?

Assuming your processors are fast enough (we recommend at least 3GHz) and you have enough cores (we recommend at least 16), here is what to expect.

Being a good network citizen

First, depending on where your 100G host lives on your network, just because you can do 100G does not mean you should do 100G. If your campus uplink is only 10G, you should not deploy a 100G host. If your campus has a single 100G uplink, you probably don't want your 100G perfSONAR host attempting to use all of the capacity of that link. Yes, your TCP flows will back off, but so will everyone else's. There are various ways to limit the speed of your 100G node. You can limit per-flow speed using flow pacing, or you may be able to configure your network switch/router optics to 10G, 25G, or 50G. For a DTN, a good way to limit the speed is to install a disk subsystem that is only capable of 40-60 Gbps.

100G perfSONAR / network test host

Even without tuning, you should be able to get 90 Gbps in back-to-back testing with enough parallel streams. Sample commands to achieve this are:

iperf3 -P 12 -c receive_host

pscheduler task throughput --parallel 12 --duration PT30S --dest receive_host

pscheduler task --tool iperf2 throughput --parallel 12 --duration PT30S --dest receive_host

Note that for iperf3 be sure to use version 3.16 or higher, as older versions are not multi-threaded. Otherwise use iperf2. As of Jan 2024, perfSONAR includes an older version of iperf3, so you'll need to update iperf3 by hand, or use iperf2.

Single stream testing will range from 15-35 Gbps, depending on how well tuned the host is, and if you are using the right combination of optimal cores on both the sender and receiver. Be sure to look at:

100G Data Transfer Node (DTN)

Performance expectations for a DTN are harder to nail down, as performance is usually disk limited, not network limited. Depending on the speed of your disk system, the number of files being transferred in parallel, and the number of parallel streams per file, we typically see between 40-80 Gbps. Getting this rate often requires transferring at least 8 files in parallel, and at least 4 parallel streams per file.

For more information see:

Hardware Selection

If you have not yet purchased servers for perfSONAR or DTNs, here are some things to keep in mind:

Campus uplink speed: Your perfSONAR/DTN hosts should not exceed the speed of your network uplink. If you know the uplink will be increased to 100G in the next few years, you should probably buy a 100G capable host, but only install 10G optics.
For a DTN, your disk subsystem throughput should roughly match your network speed. There is no point putting a 100G NIC in a host where the storage is only 1-2 GBytes/sec.

For more information on hardware selection see:

Questions?

If you have questions on these topics, feel free to contact the ESnet Science Engagement Team at engage@es.net or the NSF EPOC project at epoc@tacc.utexas.edu.