Network Performance Expectations
A first step is to try to establish a baseline to compare your throughput to the maximum possible throughput. However, determining the potential throughput of your end-to-end path can be tricky. Your local network administrator can tell you about your site capacity. Occasionally running a traceroute will tell you about other network segments if the router interface names include the capacity.
First verify that the host tuning on both the sender and receiver hosts has done properly.
If you can log into both the sending and receiving host, the easiest way to check your network throughput is using the iperf3 tool. Here are some good default settings to start with.
# try 1 stream
iperf3 -c hostname
# try a few streams
iperf3 -c hostname -P4
This gives you an upper limit on your bulk file transfer network throughput.
If the iperf3 throughput is much lower than you expect, If both hosts are properly configured and performance is still low, there is likely packet loss. Packet loss is typically due to one of the following factors: congestion, dirty fiber connectors, under-powered firewalls, or under-buffered switches. This can be tricky to diagnose. Contact your local network expert, or contact ESnet for help.
Note that it is also important to know what disk performance you should expect. It doesn't matter if you have a clean 10Gbps network path you are using slow disks. Writing to disk is typically slower that reading from disk, so try to measure the write speed on your receive host.