ping

For some types of network issues, pinging each router in the path can help identify the source of packet loss.

You'll need to use the -i0 option to send ICMP packets as fast as possible, and to do this you must be user 'root'.

Sample command: 

# ping -q -i0 -c 256 -l 4096 -s 1450 -S 10485760 -W 1 hostname

For example, use traceroute to find the hops in the path:

traceroute --icmp hostname
traceroute to hostname (198.124.42.201), 30 hops max, 60 byte packets  
1  ornlrcr5-ornlpt1.es.net (198.124.238.153)  0.692 ms  1.071 ms  1.363 ms  
2  ornl-ip-a-ornlcr5.es.net (198.124.192.118)  0.338 ms  0.404 ms  0.468 ms  
3  198.124.42.188 (198.124.42.188)  0.461 ms  0.452 ms  0.513 ms  
4  198.124.42.201 (198.124.42.201)  0.349 ms  0.340 ms  0.331 ms

Test to each hop to see where the loss begins:

# ping -q -i0 -c 256 -l 4096 -s 1450 -S 10485760 -W 1 hostname

256 packets transmitted, 256 received, 0% packet loss, time 1ms

# ping -q -i0 -c 256 -l 4096 -s 1450 -S 10485760 -W 1 hostname
 
256 packets transmitted, 256 received, 0% packet loss, time 1ms 

# ping -q -i0 -c 256 -l 4096 -s 1450 -S 10485760 -W 1 hostname

256 packets transmitted, 103 received, 59% packet loss, time 2ms

Clearly the packet loss is at hop 3.

Further investigation revealed that the 198.124.42.188 hop is a Cisco 3925 router with an HWIC Gigabit Ethernet interface.  That interface only has an 800 megabit/second data path to the forwarding engine, and the receive packet buffer isn’t very deep, which explains the packet loss issue.

Many thanks to Bill Nickless, PNNL, for this example.