fasterdata.es.netfasterdata.es.netESnet Network Performance Knowledge Base

ESnet Data Transfer Nodes

Running tests to ESnet Data Transfer Nodes (DTNs)

ESnet has deployed a set of test hosts for high-speed disk-to-disk testing. Anyone on an R&E network anywhere in the world can use these hosts for anonymous GridFTP access. These hosts are capable saturating a 10Gbps network reading from disk. These hosts are an example of a "Data Transfer Node", or DTN, for use in a Science DMZ.

These hosts are: 

  • anl-diskpt1.es.net / anl-diskpt1-v6.es.net: Near Chicago, IL
  • bnl-diskpt1.es.net / bnl-diskpt1-v6.es.net: Near NY, NY (Host is down for maintenance)
  • lbl-diskpt1.es.net / lbl-diskpt1-v6.es.net: Berkeley, CA

Globus Service Access

The test hosts are also available via the Globus Service.  They are configured for anonymous, read-only access.

  • anl-diskpt1.es.net is registered as the endpoint esnet#anl-diskpt1
  • bnl-diskpt1.es.net is registered as the endpoint esnet#bnl-diskpt1 (Host is down for maintenance)
  • lbl-diskpt1.es.net is registered as the endpoint esnet#lbl-diskpt1

Sample GridFTP test commands


If you don't have globus-url-copy installed, please refer to the GridFTP Quick Start Guide

#make sure you can connect to server
globus-url-copy -list ftp://lbl-diskpt1.es.net:2811/data1/
# copy 1G file 
globus-url-copy -vb -fast ftp://lbl-diskpt1.es.net:2811/data1/1G.dat file:///tmp/test.out
# copy 1G file using 4 parallel streams
globus-url-copy -vb -fast -p 4 ftp://lbl-diskpt1.es.net:2811/data1/1G.dat file:///tmp/test.out
# write to /dev/null
globus-url-copy -vb -fast -p 4 ftp://lbl-diskpt1.es.net:2811/data1/1G.dat file:///dev/null
# read from /dev/zero
globus-url-copy -vb -fast -p 4 -len 1G ftp://lbl-diskpt1.es.net:2811/dev/zero file:///tmp/t.out

Each host has three 1.5 TB disk arrays, mounted as /data1, /data2, and /data3. /data1 and /data2 host test data sets, while /data3 is typically used for ad-hoc testing by ESnet staff. The following test files are available on each server, and are generated using "/dev/urandom":

/data1/100M.dat, /data1/1G.dat, /data1/10G.dat, /data1/50G.dat, /data1/100G.dat
/data2/100M.dat, /data2/1G.dat, /data2/10G.dat, /data2/50G.dat, /data2/100G.dat

In addition, there are currently two data sets composed of multiple files in a directory structure. These data sets are for testing multi-file transfers. The data sets each contain directories a through z. Each of these directories contains directories a through z. Each leaf directory contains data files named for their place in the directory structure. So, a-a-1M.dat is a 1MB data file in the data set with path 5GB-in-small-files/a/a/a-a-1M.dat.

The data sets are:

/data1/5GB-in-small-files - 1MB, 2MB, and 5MB files in each leaf directory
/data1/66GB-in-small-files - 10MB, 20MB, 30MB, and 40MB files in each leaf directory
/data2/5GB-in-small-files - 1MB, 2MB, and 5MB files in each leaf directory
/data2/66GB-in-small-files - 10MB, 20MB, 30MB, and 40MB files in each leaf directory

Sample commands for copying the complete data sets:

globus-url-copy -vb -p 1 -fast -r ftp://lbl-diskpt1.es.net:2811/data1/5GB-in-small-files/ \
file:///some/path/5GB-in-small-files/ globus-url-copy -vb -p 1 -fast -r ftp://lbl-diskpt1.es.net:2811/data1/66GB-in-small-files/ \
file:///some/path/665GB-in-small-files/ globus-url-copy -vb -p 4 -fast -r ftp://lbl-diskpt1.es.net:2811/data1/5GB-in-small-files/ \
file:///some/path/5GB-in-small-files/ globus-url-copy -vb -p 4 -fast -r ftp://lbl-diskpt1.es.net:2811/data1/66GB-in-small-files/ \
file:///some/path/665GB-in-small-files/ globus-url-copy -vb -p 1 -fast -r ftp://lbl-diskpt1.es.net:2811/data2/5GB-in-small-files/ \
file:///some/path/5GB-in-small-files/ globus-url-copy -vb -p 1 -fast -r ftp://lbl-diskpt1.es.net:2811/data2/66GB-in-small-files/ \
file:///some/path/665GB-in-small-files/

These test hosts are configured to only 1 test allowed at a time. If you get this error: "421 Service busy: Connection limit exceeded. Please try again later. ", then someone else is running a test, and you should try later.

Host Tuning

It is important to make sure that your host is properly tuned for maximum TCP performance on the WAN. You should verify that htcp or cubic, and not reno, is the default TCP congestion control algorithm, and that the maximum TCP buffers are big enough for your paths of interest. For more information see the Host Tuning Section.  

Sample Throughput Results

Running disk to memory tests between the Berkeley host and the Chicago host using jumbo frames (MTU=9KB) gives these results:

1 stream  440 MB/sec  (3.5 Gbps)
4 streams,  1050 MB/sec  (8.4 Gbps)
8 streams, 1115 MB/sec  (8.9 Gbps)

Firewall Issues

Note: Sites with firewalls will need to open the ports used by GridFTP (2811 and 50000-51000) and bwctl/iperf (4823 and 5001-5050). If you have a firewall that blocks outgoing connections, you may need to set the environment variable GLOBUS_TCP_SOURCE_RANGE as well. For more information see the Globus GridFTP client firewall information guide.

IPV6 Testing

These hosts also have IPV6 addresses you can use for testing:

  •   lbl-diskpt1-v6, anl-diskpt1-v6.es.net, bnl-diskpt1-v6.es.net 
  • You'll need to add the option "-ipv6" to your globus-url-copy command line to connect to these addresses.

    Additional Information

    For details on the hardware and/or software configuration of ESnet I/O test systems, email BLTierney@es.net and Dart@es.net .