Menu
Home » Data Transfer Nodes » Hardware Selection » Storage Subsystem Selection
Data Transfer Nodes

Storage Subsystem Selection

There are often questions regarding "which storage is the right choice", or "how should storage be configured correctly".  In our examples, the ESnet reference implementation mentions a DTN using RAID6 using on-board (not networked) storage.  

The "right" configuration for storage on a DTN depends on your application, and other context (needs, budget, etc.).  For example, if all that is needed is a short-term cache, using JBOD mode or RAID0 (no redundancy, best performance) is a good choice.  For this particular use case the DTN isn't responsible for custodial storage for the data, e.g. if the filesystem crashes, the cache can be re-loaded from the authoritative source.

If the DTN must provide reliable storage, then it's probably wise to use a configuration like RAID5, RAID6, or RAID10 (e.g. RAID1+0: a RAID0 volume composed of two or more RAID1 mirrors rather than two or more individual disks).  

This is just one possible set of tradeoffs between performance, cost, and resilience.  There are two main considerations for DTN storage:

  • Networked storage
    • Many who have DTNs use IB cards to get to a high speed parallel file system.  This prevents the DTN from needing to be anything other than a highly connected and tuned client of the storage but not necessarily to worry about resiliency.
  • Local storage
    • Two options here, either use hardware RAID cards to get to some direct attached storage off a server/DTN, or use software to gang drives in the node into RAID.  Some installations are using large servers that have a large number of hard drives in them and used Linux RAID capabilities to give some very impressive performance while maintaining low cost and good resiliency.  RAID5 or RAID6 is often used in this case.  


It is often the case that for large-scale infrastructure, the DTNs are often completely distinct from the set of servers that implement the filesystem.  The DTN just runs the data transfer application and is a client of the filesystem.  This architecture has a variety of advantages - it is possible to write a distinct security policy for DTNs (a la Science DMZ), the servers that implement the filesystem is not exposed to to users directly, and tools like Globus allow a single endpoint out of multiple DTNs which is good for performance, resiliency, and ease of management.

Lastly, when using/installing the hardware pay close attention to drivers that are used on the software side.  It is often the case that the stock distro driver/firmware is woefully out of date.  This is especially true of high-performance network cards, and is also true for disk controllers as well.  Follow documentation from the vendor on if its appropriate to download/maintain special drivers independent of the operating system defaults.

Disk

SATA disks historically have been cheaper and higher capacity, while SAS disks typically have been the fastest. However these technologies have been converging, and with SATA 3.1 is less true. NVMe backplanes are now also a viable option for high-performance I/O using all-flash storage. HPE has a document describing storage interfaces that might help you decide what is best for you.

SSD

SSD storage costs much more than traditional hard drives (HD), but are much faster. They come in different styles:

  • PCIe card: some vendors build PCI cards with SSD. These are the fastest type of SSD: up to several GBytes/sec per card. Note that this type of SSD is typically not hot-swapable.
  • HD replacement: several vendors now sell SSD-based drives that have the same form factor as traditional drives such as SAS and SATA. The downside to this approach is that performance is limited by the RAID controller, and not all controllers work well with SSD. Be sure that your controller is “SSD capable”.

Note that the price of SSD, particularly HD replacement style, is coming down quickly, so a SSD-based solution may be worth considering for your DTNs. A useful articles on SSDs reliability available here.

If there is a desire to get more than 20Gbps disk performance out of a single machine, the use of SSDs will be necessary.  The best performance will come from the use of NVMe drives. Note that each variety of SSD is optimized for a different type of workflow, and possesses a myriad of tuning options to get peak performance.  Repeated P/E (program-erase) cycles will shorten the life of an SSD, and it is recommended that certain tuning parameters be applied to help mitigate this.

Selecting an SSD can be a challenging task due to the every changing nature of the market.  It is suggested that you consult a recent guide (e.g. http://www.fastestssd.com/featured/ssd-rankings-the-fastest-solid-state-drives/) as you evaluate speed and reliability factors.  

RAID Controller

Be sure your RAID controller has the following:

  • 1GB of on-board cache
  • PCIe Gen3/4 support
  • dual-core RAID-on-Chip (ROC) processor if you will have more than 8 drives

One example of a RAID card that satisfies these criteria is the Areca ARC-1882i.

More details on various RAID controller features can be found here.