Menu

ESnet Data Transfer Nodes

Running tests to ESnet Data Transfer Nodes (DTNs)

ESnet has deployed a set of test hosts for high-speed disk-to-disk testing. Anyone on an R&E network anywhere in the world can use these hosts for anonymous GridFTP access. These hosts are capable saturating a 10Gbps network reading from disk. These hosts are an example of a "Data Transfer Node", or DTN, for use in a Science DMZ.

We recently upgraded these hosts - they have faster motherboards and processors, faster disk controllers, and more space.  The highlights of the upgrade are:

  • Over 8Gbps from a single filesystem namespace onto the network
  • Expanded range of test files and directory structures
  • Only one data directory (currently /storage/data1/gridftp or /data1 - they point to the same place)

The test DTN hosts are:

  • anl-diskpt1.es.net / anl-diskpt1-v6.es.net: Near Chicago, IL
  • bnl-diskpt1.es.net / bnl-diskpt1-v6.es.net: Near NYC, NY (host will be down intermittently for maintenance this week)
  • lbl-diskpt1.es.net / lbl-diskpt1-v6.es.net: Berkeley, CA
  • cern-diskpt1.es.net / cern-diskpt1-v6.es.net: Geneva, Switzerland 

 

Globus Service Access

The test hosts are also available via the Globus Transfer service.  They are configured for anonymous, read-only access.

  • anl-diskpt1.es.net is registered as the endpoint ESnet test DTN at ANL
  • bnl-diskpt1.es.net is registered as the endpoint ESnet test DTN at BNL  (host will be down intermittently for maintenance this week)
  • lbl-diskpt1.es.net is registered as the endpoint ESnet test DTN at LBL
  • cern-diskpt1.es.net is registered as the endpoint ESnet test DTN at CERN

 

Test data sets

Each host has a high-performance disk array, mounted as /data1. The following test files are available on each server, and are generated using "/dev/urandom" (the size is what you would expect from reading the filename):

/data1/1M.dat, /data1/10M.dat, /data1/50M.dat, /data1/100M.dat,
/data1/1G.dat, /data1/10G.dat, /data1/50G.dat, /data1/100G.dat, /data1/500G.dat

In addition, there are currently several data sets composed of multiple files in a directory structure. These data sets are for testing multi-file transfers. The data sets each contain directories a through y. Each of these directories contains directories a through y. Each leaf directory contains data files named for their place in the directory structure. So, a-a-1M.dat is a 1,000,000 byte data file in the data set with path 5GB-in-small-files/a/a/a-a-1M.dat. 

The structure and composition of the test data sets is designed so that the relative impact of metadata operations (e.g. file and directory creation) and data transfer (moving the data file contents) can be measured.  The directory structures are identical, but the file sizes vary.  This means that one can transfer the 5MB-in-tiny-files directory structure to measure the transfer overhead of file and directory creation performance, and one of the larger structures to measure the increase in data transfer time which is due to the increased file size.

The test data sets are:

/data1/5MB-in-tiny-files - 1KB, 2KB, and 5KB files in each leaf directory
/data1/5GB-in-small-files - 1MB, 2MB, and 5MB files in each leaf directory /data1/50GB-in-medium-files - 10MB, 20MB, and 50MB files in each leaf directory /data1/500GB-in-large-files - 100MB, 200MB, and 500MB files in each leaf directory

There are also four directories containing climate model data of different file sizes. Each data set is about 240GB in total size. These data sets are intended for use by the ICNWG member sites, though others are welcome to use them as well.  The Climate-Small data set has an internal directory structure, where the other three do not (Climate-Small is a portion of the CORDEX data set, with its internal directory structure left intact and pruned to be about 240GB in size). The climate data set composition is as follows:

/data1/Climate-Huge - two files, each ~120GB
/data1/Climate-Large - 10 files, each 21.5GB plus one 28.8GB file
/data1/Climate-Medium - 117 files, ranging in size from 1.2GB to 6GB
/data1/Climate-Small - 1,496 files, ranging in size from 29MB to 425MB




Sample GridFTP test commands

In addition to Globus Transfer access, the data sets are provided by GridFTP server for anonymous, read-only access by command-line globus-url-copy.

If you don't have globus-url-copy installed, please refer to the GridFTP Quick Start Guide

#make sure you can connect to server
globus-url-copy -list ftp://lbl-diskpt1.es.net:2811/data1/
# copy 1G file 
globus-url-copy -vb -fast ftp://lbl-diskpt1.es.net:2811/data1/10G.dat file:///tmp/test.out
# copy 1G file using 4 parallel streams
globus-url-copy -vb -fast -p 4 ftp://lbl-diskpt1.es.net:2811/data1/10G.dat file:///tmp/test.out
# write to /dev/null
globus-url-copy -vb -fast -p 4 ftp://lbl-diskpt1.es.net:2811/data1/10G.dat file:///dev/null
# read from /dev/zero
globus-url-copy -vb -fast -p 4 -len 1G ftp://lbl-diskpt1.es.net:2811/dev/zero file:///tmp/t.out
# Option to use UDT instead of TCP
globus-url-copy -vb -udt ftp://lbl-diskpt1.es.net:2811/data1/10G.dat file:///dev/null

 

Sample commands for copying the complete data sets (these use the Berkeley DTN - substitute the other DTNs as needed):

# Copy using one stream only to test single-stream disk-to-disk performance
globus-url-copy -vb -p 1 -fast -r ftp://lbl-diskpt1.es.net:2811/data1/5GB-in-small-files/ \
file:///some/path/5GB-in-small-files/
#
globus-url-copy -vb -p 1 -fast -r ftp://lbl-diskpt1.es.net:2811/data1/50GB-in-medium-files/ \
file:///some/path/50GB-in-medium-files/ #
# Copy using 4 parallel streams
globus-url-copy -vb -p 4 -fast -r ftp://lbl-diskpt1.es.net:2811/data1/5GB-in-small-files/ \
file:///some/path/5GB-in-small-files/
#
globus-url-copy -vb -p 4 -fast -r ftp://lbl-diskpt1.es.net:2811/data1/50GB-in-medium-files/ \
file:///some/path/50GB-in-medium-files/
#
# Copy the big data set using 8 parallel streams
# (make sure your performance is good before doing this one!)
globus-url-copy -vb -p 8 -fast -r ftp://lbl-diskpt1.es.net:2811/data1/500GB-in-large-files/ \
file:///some/path/500GB-in-large-files/

Host Tuning

It is important to make sure that your host is properly tuned for maximum TCP performance on the WAN. You should verify that htcp or cubic, and not reno, is the default TCP congestion control algorithm, and that the maximum TCP buffers are big enough for your paths of interest. For more information see the Host Tuning Section.  

Sample Throughput Results

Running memory to memory tests between the ANL host and the BNL host using jumbo frames (MTU=9000 bytes) gives these results:

  • 1 stream  1.0 GB/sec  (8.0 Gbps)
  • 4 streams,  1.1 GB/sec  (8.8 Gbps) 
  • 1 stream UDT, 220 MB/sec  (1.7 Gbps) (UDT is slower than TCP on a clean network, but will be faster than TCP on a network with packet loss)

Running disk to memory tests between the ANL host and the BNL host using jumbo frames (MTU=9000 bytes) gives only slightly lower results:

  • 4 streams,  950 MB/sec  (7.6 Gbps)

The hardware configuration for these hosts is available here.

Firewall Issues

Note: Sites with firewalls will need to open the ports used by GridFTP (2811 and 50000-51000) and bwctl/iperf (4823 and 5001-5050). Depending on your firewall configuration you may need to set the environment variable GLOBUS_TCP_SOURCE_RANGE to 50000-51000 as well. For more information see the Globus GridFTP client firewall information guide.

IPV6 Testing

These hosts also have IPV6 addresses you can use for testing:

  •   lbl-diskpt1-v6, anl-diskpt1-v6.es.net, bnl-diskpt1-v6.es.net, cern-272-diskpt1-v6.es.net

You'll need to add the option "-ipv6" to your globus-url-copy command line to connect to these addresses.

Additional Information

For details on the hardware and/or software configuration of ESnet I/O test systems, email BLTierney@es.net and Dart@es.net .