ESnet Data Transfer Nodes
Running tests to ESnet Data Transfer Nodes (DTNs)
ESnet has deployed a set of test hosts for high-speed disk-to-disk testing. Anyone on an R&E network anywhere in the world can use these hosts for anonymous GridFTP access. These hosts are capable saturating a 10Gbps network reading from disk. These hosts are an example of a "Data Transfer Node", or DTN, for use in a Science DMZ.
We recently upgraded these hosts - they have faster motherboards and processors, faster disk controllers, and more space. The highlights of the upgrade are:
- Over 8Gbps from a single filesystem namespace onto the network
- Expanded range of test files and directory structures
- Only one data directory (currently /storage/data1/gridftp or /data1 - they point to the same place)
The test DTN hosts are:
- anl-diskpt1.es.net / anl-diskpt1-v6.es.net: Near Chicago, IL
- bnl-diskpt1.es.net / bnl-diskpt1-v6.es.net: Near NYC, NY (down for maintenance)
- lbl-diskpt1.es.net / lbl-diskpt1-v6.es.net: Berkeley, CA
- cern-diskpt1.es.net / cern-diskpt1-v6.es.net: Geneva, Switzerland
Globus Service Access
The test hosts are also available via the Globus Transfer service. They are configured for anonymous, read-only access.
- anl-diskpt1.es.net is registered as the endpoint ESnet test DTN at ANL
- bnl-diskpt1.es.net is registered as the endpoint ESnet test DTN at BNL (down for maintenance)
- lbl-diskpt1.es.net is registered as the endpoint ESnet test DTN at LBL
- cern-diskpt1.es.net is registered as the endpoint ESnet test DTN at CERN
Test data sets
Each host has a high-performance disk array, mounted as /data1. The following test files are available on each server, and are generated using "/dev/urandom" (the size is what you would expect from reading the filename):
/data1/1M.dat, /data1/10M.dat, /data1/50M.dat, /data1/100M.dat,
/data1/1G.dat, /data1/10G.dat, /data1/50G.dat, /data1/100G.dat, /data1/500G.dat
In addition, there are currently several data sets composed of multiple files in a directory structure. These data sets are for testing multi-file transfers. The data sets each contain directories a through y. Each of these directories contains directories a through y. Each leaf directory contains data files named for their place in the directory structure. So, a-a-1M.dat is a 1,000,000 byte data file in the data set with path 5GB-in-small-files/a/a/a-a-1M.dat.
The structure and composition of the test data sets is designed so that the relative impact of metadata operations (e.g. file and directory creation) and data transfer (moving the data file contents) can be measured. The directory structures are identical, but the file sizes vary. This means that one can transfer the 5MB-in-tiny-files directory structure to measure the transfer overhead of file and directory creation performance, and one of the larger structures to measure the increase in data transfer time which is due to the increased file size.
The test data sets are:
/data1/5MB-in-tiny-files - 1KB, 2KB, and 5KB files in each leaf directory
/data1/5GB-in-small-files - 1MB, 2MB, and 5MB files in each leaf directory /data1/50GB-in-medium-files - 10MB, 20MB, and 50MB files in each leaf directory /data1/500GB-in-large-files - 100MB, 200MB, and 500MB files in each leaf directory
There are also four directories containing climate model data of different file sizes. Each data set is about 240GB in total size. These data sets are intended for use by the ICNWG member sites, though others are welcome to use them as well. The Climate-Small data set has an internal directory structure, where the other three do not (Climate-Small is a portion of the CORDEX data set, with its internal directory structure left intact and pruned to be about 240GB in size). The climate data set composition is as follows:
/data1/Climate-Huge - two files, each ~120GB /data1/Climate-Large - 10 files, each 21.5GB plus one 28.8GB file /data1/Climate-Medium - 117 files, ranging in size from 1.2GB to 6GB
/data1/Climate-Small - 1,496 files, ranging in size from 29MB to 425MB
Sample GridFTP test commands
In addition to Globus Transfer access, the data sets are provided by GridFTP server for anonymous, read-only access by command-line globus-url-copy.
If you don't have globus-url-copy installed, please refer to the GridFTP Quick Start Guide
#make sure you can connect to server globus-url-copy -list ftp://lbl-diskpt1.es.net:2811/data1/ # copy 1G file globus-url-copy -vb -fast ftp://lbl-diskpt1.es.net:2811/data1/10G.dat file:///tmp/test.out # copy 1G file using 4 parallel streams globus-url-copy -vb -fast -p 4 ftp://lbl-diskpt1.es.net:2811/data1/10G.dat file:///tmp/test.out # write to /dev/null globus-url-copy -vb -fast -p 4 ftp://lbl-diskpt1.es.net:2811/data1/10G.dat file:///dev/null # read from /dev/zero globus-url-copy -vb -fast -p 4 -len 1G ftp://lbl-diskpt1.es.net:2811/dev/zero file:///tmp/t.out # Option to use UDT instead of TCP globus-url-copy -vb -udt ftp://lbl-diskpt1.es.net:2811/data1/10G.dat file:///dev/null
Sample commands for copying the complete data sets (these use the Berkeley DTN - substitute the other DTNs as needed):
# Copy using one stream only to test single-stream disk-to-disk performance
globus-url-copy -vb -p 1 -fast -r ftp://lbl-diskpt1.es.net:2811/data1/5GB-in-small-files/ \
globus-url-copy -vb -p 1 -fast -r ftp://lbl-diskpt1.es.net:2811/data1/50GB-in-medium-files/ \
# Copy using 4 parallel streams
globus-url-copy -vb -p 4 -fast -r ftp://lbl-diskpt1.es.net:2811/data1/5GB-in-small-files/ \
globus-url-copy -vb -p 4 -fast -r ftp://lbl-diskpt1.es.net:2811/data1/50GB-in-medium-files/ \
# Copy the big data set using 8 parallel streams
# (make sure your performance is good before doing this one!)
globus-url-copy -vb -p 8 -fast -r ftp://lbl-diskpt1.es.net:2811/data1/500GB-in-large-files/ \
It is important to make sure that your host is properly tuned for maximum TCP performance on the WAN. You should verify that htcp or cubic, and not reno, is the default TCP congestion control algorithm, and that the maximum TCP buffers are big enough for your paths of interest. For more information see the Host Tuning Section.
Sample Throughput Results
Running memory to memory tests between the ANL host and the BNL host using jumbo frames (MTU=9000 bytes) gives these results:
- 1 stream 1.0 GB/sec (8.0 Gbps)
- 4 streams, 1.1 GB/sec (8.8 Gbps)
- 1 stream UDT, 220 MB/sec (1.7 Gbps) (UDT is slower than TCP on a clean network, but will be faster than TCP on a network with packet loss)
Running disk to memory tests between the ANL host and the BNL host using jumbo frames (MTU=9000 bytes) gives only slightly lower results:
- 4 streams, 950 MB/sec (7.6 Gbps)
The hardware configuration for these hosts is available here.
Note: Sites with firewalls will need to open the ports used by GridFTP (2811 and 50000-51000) and bwctl/iperf (4823 and 5001-5050). Depending on your firewall configuration you may need to set the environment variable GLOBUS_TCP_SOURCE_RANGE to 50000-51000 as well. For more information see the Globus GridFTP client firewall information guide.
These hosts also have IPV6 addresses you can use for testing:
- lbl-diskpt1-v6, anl-diskpt1-v6.es.net, bnl-diskpt1-v6.es.net, cern-272-diskpt1-v6.es.net
You'll need to add the option "-ipv6" to your globus-url-copy command line to connect to these addresses.
For details on the hardware and/or software configuration of ESnet I/O test systems, email BLTierney@es.net and Dart@es.net .