Standard Unix Tools

Here are some useful Unix open source tools

Standard Unix tools:

These tools come pre-installed on many Unix distributions.

Limitations: no parallel stream support.

  • curl: curl is a command line tool for transferring data with URL syntax, supporting DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, TELNET and TFTP. curl supports SSL certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, proxies, cookies, user+password authentication (Basic, Digest, NTLM, Negotiate, kerberos...), file transfer resume, proxy tunneling and more .
  • wget: wget is a command line too for transferring URL via HTTP, HTTPS and FTP,

Tools that can be added using yum, apt-get, port, fink, etc:

  • aria2: Next generation download utility

Supports HTTP/HTTPS, FTP, SFTP, BitTorrent and Metalink. aria2 can be manipulated via built-in JSON-RPC and XML-RPC interfaces.  Allows for multiple connections. 

Sample commands:

  aria2c -x2 http://a/f.iso
  • lftp: Supports parallel file transfer, socket tuning, HTTP, FTP, SFTP, bittorent transfers, and more.

Limitations: non intuitive syntax

Sample commands:

  lftp -e 'set net:socket-buffer 4000000; pget -n 4 http://site/path/file; quit'
lftp -e 'set net:socket-buffer 4000000; pget -n 4 ftp://site/path/file; quit'
  • axel: simple parallel accelerator for HTTP and FTP.

Limitations: no authentication support

Sample commands:

axel -n 4 http://site/file
axel -n 4 ftp://site/file