ESnet 40G perfSONAR Host

ESnet has deployed a perfSONAR host in Boston, MA with a 40Gbps NIC.

The specs for this host are:

    • Super X10SL7-F Motherboard
    • 1 Intel Xeon E3-1275 v3 3.5GHz (Turbo 3.9GHz) Quad Cores
    • Mellanox ConnectX-3 40G ethernet NIC
    • 32GB DDRIII 1600 ECC Only Certified Memory
    • Seagate 1TB SAS 6GB/s, 7200RPM ST1000NM002

The tuning settings for this host are:

Add to /etc/sysctl/conf
# increase TCP max buffer size setable using setsockopt() net.core.rmem_max = 536870912 net.core.wmem_max = 536870912 # increase Linux autotuning TCP buffer limit net.ipv4.tcp_rmem = 4096 87380 268435456 net.ipv4.tcp_wmem = 4096 65536 268435456 # don't cache ssthresh from previous connection net.ipv4.tcp_no_metrics_save = 1   Add to /etc/rc.d/rc.local:

Note that this host only has a single CPU socket, so IRQ and core selection tuning is not required.

To run tests to this host, install the perfSONAR tool suite:

 rpm -hUv
yum -y install perfsonar-tools

Then run:

   bwctl -c -T iperf3 -w 256M -t30