fasterdata.es.netfasterdata.es.netESnet Network Performance Knowledge Base
Performance Testing

ESnet IO Testers

Running tests to ESnet I/O Test Hosts

ESnet has deployed a set of I/O Test hosts for high-speed disk-to-disk testing. These hosts are capable of 35 Gbps network throughput (using 4 10GE NICS, only 1 of which is currently connected to ESnet), 16 Gbps disk read, and 12 Gbps disk write. They are running bwctl for network testing, and run an anonymous read-only GridFTP server for disk-to-disk testing. These services are available to anyone on an R&E network anywhere in the world.

These hosts are:

  • lbl-diskpt1.es.net / lbl-diskpt1-v6.es.net: Berkeley, CA
  • anl-diskpt1.es.net / anl-diskpt1-v6.es.net: Near Chicago, IL
  • bnl-diskpt1.es.net / bnl-diskpt1-v6.es.net: Near NY, NY

These hosts are an example of a "Data Transfer Node", or DTN. We describe an example of a DTN configuration here.

Sample GridFTP test commands


If you don't have globus-url-copy installed, please refer to the GridFTP Quick Start Guide

#make sure you can connect to server
globus-url-copy -list ftp://lbl-diskpt1.es.net:2811/data1/
# copy 1G file 
globus-url-copy -vb -fast ftp://lbl-diskpt1.es.net:2811/data1/1G.dat file:///tmp/test.out
# copy 1G file using 4 parallel streams
globus-url-copy -vb -fast -p 4 ftp://lbl-diskpt1.es.net:2811/data1/1G.dat file:///tmp/test.out
# write to /dev/null
globus-url-copy -vb -fast -p 4 ftp://lbl-diskpt1.es.net:2811/data1/1G.dat file:///dev/null
# read from /dev/zero
globus-url-copy -vb -fast -p 4 -len 1G ftp://lbl-diskpt1.es.net:2811/dev/zero file:///tmp/t.out

Each host has three 1.5 TB disk arrays, mounted as /data1, /data2, and /data3. /data1 and /data2 host test data sets, while /data3 is typically used for ad-hoc testing by ESnet staff. The following test files are available on each server, and are generated using "/dev/urandom":

/data1/100M.dat, /data1/1G.dat, /data1/10G.dat, /data1/50G.dat, /data1/100G.dat
/data2/100M.dat, /data2/1G.dat, /data2/10G.dat, /data2/50G.dat, /data2/100G.dat

In addition, there are currently two data sets composed of multiple files in a directory structure. These data sets are for testing multi-file transfers. The data sets each contain directories a through z. Each of these directories contains directories a through z. Each leaf directory contains data files named for their place in the directory structure. So, a-a-1M.dat is a 1MB data file in the data set with path 5GB-in-small-files/a/a/a-a-1M.dat.

The data sets are:

/data1/5GB-in-small-files - 1MB, 2MB, and 5MB files in each leaf directory
/data1/66GB-in-small-files - 10MB, 20MB, 30MB, and 40MB files in each leaf directory
/data2/5GB-in-small-files - 1MB, 2MB, and 5MB files in each leaf directory
/data2/66GB-in-small-files - 10MB, 20MB, 30MB, and 40MB files in each leaf directory

Sample commands for copying the complete data sets:

globus-url-copy -vb -p 1 -fast -r ftp://lbl-diskpt1.es.net:2811/data1/5GB-in-small-files/ \
     file:///some/path/5GB-in-small-files/
globus-url-copy -vb -p 1 -fast -r ftp://lbl-diskpt1.es.net:2811/data1/66GB-in-small-files/ \
     file:///some/path/665GB-in-small-files/

globus-url-copy -vb -p 4 -fast -r ftp://lbl-diskpt1.es.net:2811/data1/5GB-in-small-files/ \
     file:///some/path/5GB-in-small-files/
globus-url-copy -vb -p 4 -fast -r ftp://lbl-diskpt1.es.net:2811/data1/66GB-in-small-files/ \
     file:///some/path/665GB-in-small-files/

globus-url-copy -vb -p 1 -fast -r ftp://lbl-diskpt1.es.net:2811/data2/5GB-in-small-files/ \
     file:///some/path/5GB-in-small-files/
globus-url-copy -vb -p 1 -fast -r ftp://lbl-diskpt1.es.net:2811/data2/66GB-in-small-files/ \
     file:///some/path/665GB-in-small-files/

These test hosts are configured to only 1 test allowed at a time. If you get this error: "421 Service busy: Connection limit exceeded. Please try again later. ", then someone else is running a test, and you should try later.

Host Tuning

It is important to make sure that your host is properly tuned for maximum TCP performance on the WAN. You should verify that htcp or cubic, and not reno, is the default TCP congestion control algorithm, and that the maximum TCP buffers are big enough for your paths of interest. For more information see the Host Tuning Section.  

Sample Throughput Results

Running disk to disk tests between the Berkeley host and the Chicago host using jumbo frames (MTU=9KB) gives these results:

1 stream  175 MB/sec  (1.4 Gbps)
4 streams,  680 MB/sec  (5.4 Gbps)
8 streams, 820 MB/sec  (6.6 Gbps)

Firewall Issues

Note: Sites with firewalls will need to open the ports used by GridFTP (2811 and 50000-51000) and bwctl/iperf (4823 and 5001-5050). If you have a firewall that blocks outgoing connections, you may need to set the environment variable GLOBUS_TCP_SOURCE_RANGE as well. For more information see the Globus GridFTP client firewall information guide.

Additional Information

For details on the hardware and/or software configuration of ESnet I/O test systems, email BLTierney@es.net and Dart@es.net .