Menu

Command Line Tools

The following are some commonly used tools for data transfer:

Tool

Summary

curl

curl is a command line tool for transferring data with URL syntax, supporting DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, TELNET and TFTP. curl supports SSL certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, proxies, cookies, user+password authentication (Basic, Digest, NTLM, Negotiate, kerberos...), file transfer resume, proxy tunneling and more.

Limitations: no parallel stream support.

wget

wget is a command line too for transferring URL via HTTP, HTTPS and FTP

Limitations: no parallel stream support.

scp/sftp/rsync

scp, sftp, and rsync are commonly used to copy data between hosts.  See this resource for more information: https://fasterdata.es.net/data-transfer-tools/scp-and-sftp/

aria2

Next generation download utility

Supports HTTP/HTTPS, FTP, SFTP, BitTorrent and Metalink. aria2 can be manipulated via built-in JSON-RPC and XML-RPC interfaces.  Allows for multiple connections. 

Sample commands:

  aria2c -x2 http://a/f.iso

lftp

Supports parallel file transfer, socket tuning, HTTP, FTP, SFTP, bittorent transfers, and more.

Limitations: non intuitive syntax

Sample commands:

  lftp -e 'set net:socket-buffer 4000000; pget -n 4 http://site/path/file; quit'
lftp -e 'set net:socket-buffer 4000000; pget -n 4 ftp://site/path/file; quit'

axel

Simple parallel accelerator for HTTP and FTP.

Limitations: no authentication support

Sample commands:

axel -n 4 http://site/file
axel -n 4 ftp://site/file

GridFTP

This tool comes from part of the Globus Toolkit.  To install GridFTP with ssh support, see our Quick Start Guide. The guide also has additional sample commands and other information.

Sample Command

globus-url-copy -p 4 sshftp://data.lbl.gov/home/mydata/myfile \
file://home/mydir/myfile

 

FDT

FDT is an Application for Efficient Data Transfers which is capable of reading and writing at disk speed over wide area networks (with standard TCP). It is written in Java, runs an all major platforms and it is easy to use. FDT is based on an asynchronous, flexible multithreaded system and is using the capabilities of the Java NIO libraries.

FDT can be used in one of these seven modes:

  • Server: java -jar fdt.jar [ OPTIONS ]
  • Client: java -jar fdt.jar [ OPTIONS ] -c <host> [file1 …]|[-fl <fileList>] -d <destinationDirectory>
  • SCP: java -jar fdt.jar [ OPTIONS ] [[[user@][host1:]]file1 [[[user@][host2:]]file2
  • Coordinator: java -jar fdt.jar [OPTIONS] -dIP <destination-ip> -dp <destination-port> -sIP <source-ip> -p <source-port> -d <destinationDirectory> [-fl <fileList>] -coord
  • List Files: java -jar fdt.jar [OPTIONS] -c <host> -ls <ls-path>
  • Agent: java -jar fdt.jar [OPTIONS] -c <host> -tp <transfer,ports,separated,with,comma> -agent
  • Session log: java -jar fdt.jar [OPTIONS] -c <host> -d <destinationDirectory> -sID <session-ID>

UDR

UDR is a replacement for rsync that used the UDP-based UDT protocol instead of TCP. It works very well for speeds up to 1-2 Gbps without tuning, and can go faster with proper tuning.

Sample Command

udr rsync -av --stats --progress /home/user/tmp/ hostname.com:/home/user/tmp
 

bbcp

This tool was developed at SLAC for Baber. More info on using bbcp is available from Caltech.

Sample Command

bbcp -P 4 -v myfile remotehost:filename