Command Line Tools

Tool

Summary

curl

curl is a command line tool for transferring data with URL syntax, supporting DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, TELNET and TFTP. curl supports SSL certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, proxies, cookies, user+password authentication (Basic, Digest, NTLM, Negotiate, kerberos...), file transfer resume, proxy tunneling and more.

Limitations: no parallel stream support.

wget

wget is a command line too for transferring URL via HTTP, HTTPS and FTP

Limitations: no parallel stream support.

scp/sftp/rsync

scp, sftp, and rsync are commonly used to copy data between hosts.  See this resource for more information: https://fasterdata.es.net/data-transfer-tools/scp-and-sftp/

aria2

Next generation download utility

Supports HTTP/HTTPS, FTP, SFTP, BitTorrent and Metalink. aria2 can be manipulated via built-in JSON-RPC and XML-RPC interfaces.  Allows for multiple connections. 

Sample commands:

  aria2c -x2 http://a/f.iso

lftp

Supports parallel file transfer, socket tuning, HTTP, FTP, SFTP, bittorent transfers, and more.

Limitations: non intuitive syntax

Sample commands:

  lftp -e 'set net:socket-buffer 4000000; pget -n 4 http://site/path/file; quit'
  lftp -e 'set net:socket-buffer 4000000; pget -n 4 ftp://site/path/file; quit'

axel

Simple parallel accelerator for HTTP and FTP.

Limitations: no authentication support

Sample commands:

axel -n 4 http://site/file
axel -n 4 ftp://site/file

More info is here.

FDT

FDT is an Application for Efficient Data Transfers which is capable of reading and writing at disk speed over wide area networks (with standard TCP). It is written in Java, runs an all major platforms and it is easy to use. FDT is based on an asynchronous, flexible multithreaded system and is using the capabilities of the Java NIO libraries.

FDT can be used in one of these seven modes:

• Server: java -jar fdt.jar [ OPTIONS ]
• Client: java -jar fdt.jar [ OPTIONS ] -c <host> [file1 …]|[-fl <fileList>] -d <destinationDirectory>
• SCP: java -jar fdt.jar [ OPTIONS ] [[[user@][host1:]]file1 [[[user@][host2:]]file2
• Coordinator: java -jar fdt.jar [OPTIONS] -dIP <destination-ip> -dp <destination-port> -sIP <source-ip> -p <source-port> -d <destinationDirectory> [-fl <fileList>] -coord
• List Files: java -jar fdt.jar [OPTIONS] -c <host> -ls <ls-path>
• Agent: java -jar fdt.jar [OPTIONS] -c <host> -tp <transfer,ports,separated,with,comma> -agent
• Session log: java -jar fdt.jar [OPTIONS] -c <host> -d <destinationDirectory> -sID <session-ID>

bbcp

This tool was developed at SLAC. More info on using bbcp is available from Caltech.

Sample Command

bbcp -P 4 -v myfile remotehost:filename