NETWORK DEBUGGING

In a recently installed VPS, I encountered a strange error when performing simple tasks such as apt-get update. It turned out to be a simple fix. There were no name servers defined in the instance and was fixed by adding two nameservers to /etc/resolv.conf.

That got me thinking about this week’s article. How do you trouble shoot network connections on your server? We will focus primarily on external connectivity from the server to the internet (though I do see a future article that talks about understanding network interfaces, setting up dhcp etc.).

Traceroute

If you are looking to trace how data is moving across the internet from your server to other locations, traceroute is the tool for you. It also helps you identify where packet loss occurs.

traceroute is available as default in all Linux flavors. However, if you need to install it,

# apt-get install traceroute    # Ubuntu/Debian

# yum install traceroute        # CentOS/RHEL

The command traceroute requires only one mandatory argument – the destination IP address or the host name you want to reach. Output is shown in terms of “hops”, i.e. the jump from one point in the network (a server/router) to the next as the sample packet of data moves from source to destination.

The output of the command shows each hop as one line entry, hops that fail are represented by * * * entries. However this does not mean that the connection stops right there. It could mean either of two things – the gateway is not configured to send a “time exceeded” message or it has set a TTL (time-to-live) that is too small for the source to receive the data.

Here is the output of a traceroute to google’s name servers.

$ traceroute 8.8.8.8

traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets

1 193.187.xxx.xxx (193.187.xxx.xxx) 0.082 ms 0.031 ms 0.026 ms

2 172.82.163.193 (172.82.163.193) 0.387 ms 0.345 ms 0.396 ms

3 69.174.10.149 (69.174.10.149) 0.353 ms 0.305 ms 0.258 ms

4 et-0-0-71.cr4-lax2.ip4.gtt.net (89.149.140.73) 0.432 ms 0.401 ms 
et-0-0-67.cr4-lax2.ip4.gtt.net (89.149.140.77) 0.313 ms

5 as15169.cr4-lax2.ip4.gtt.net (216.221.156.186) 0.369 ms 0.337 ms 0.350 ms

6 108.170.247.161 (108.170.247.161) 0.351 ms 108.170.247.225 
(108.170.247.225) 0.482 ms 0.510 ms

7 108.177.3.233 (108.177.3.233) 0.498 ms 108.177.3.225 (108.177.3.225) 0.500 ms 
216.239.49.237 (216.239.49.237) 0.381 ms

8 google-public-dns-a.google.com (8.8.8.8) 0.263 ms 0.374 ms 0.359 ms

You can also provide a domain name instead of an IP address (saves you time to do the lookup)

Here is the search to hudsonvalleyhost’s main site.

$ traceroute hudsonvalleyhost.com

traceroute to hudsonvalleyhost.com (104.18.50.193), 30 hops max, 60 byte packets

1 193.187.xxx.xxx (193.187.xxx.xxx) 0.071 ms 0.030 ms 0.028 ms

2 172.82.163.193 (172.82.163.193) 0.349 ms 0.415 ms 0.400 ms

3 ae1-79.cr2-lax2.ip4.gtt.net (69.174.10.149) 0.294 ms 0.276 ms 1.349 ms

4 et-0-0-31.cr4-lax1.ip4.gtt.net (89.149.187.102) 14.169 ms 14.130 ms 14.092 ms

5 ip4.gtt.net (69.174.7.78) 1.093 ms 27.459 ms 27.422 ms

6 104.18.50.193 (104.18.50.193) 0.344 ms 0.383 ms 0.343 ms

As you can see, each hop is also looked-up to get a qualified domain name. This involves a query to a nameserver. If you wish to avoid this, you can execute the same command with a -n option

$ traceroute hudsonvalleyhost.com -n

traceroute to hudsonvalleyhost.com (104.18.51.193), 30 hops max, 60 byte packets

1 193.187.xxx.xxx 0.070 ms 0.029 ms 0.031 ms

2 172.82.163.193 7.837 ms 9.771 ms 9.778 ms

3 69.174.10.149 2.003 ms 1.944 ms 0.270 ms

4 89.149.187.102 0.676 ms 0.628 ms 0.585 ms

5 69.174.7.78 0.915 ms 2.622 ms 15.441 ms

6 104.18.51.193 0.357 ms 0.356 ms 0.333 ms

This is much quicker and recommended if you don’t need to identify the gateway names. There are other options available with traceroute, the complete list available from the traceroute –help command.

MTR

mtr adds more detail to traceroute as it provides an average response time between each hop and packet loss by hop

This handy tool may need to be installed.

# apt-get install mtr    # Ubuntu/Debian

# yum install mtr        # CentOS/RHEL

The simplest execution of mtr is

$ mtr --report IP-ADDRESS

Here is the report for Google’s name server again

$ mtr --report 8.8.8.8

Start: 2019-03-22T14:22:55-0400

HOST: abytecurious.bid           Loss%   Snt  Last   Avg   Best   Wrst   StDev

1.|-- 10.8.54.85                  0.0%   10   1.3   11.9   1.2   51.9   20.7

2.|-- 10.8.47.49                  0.0%   10   0.7    0.7   0.5   0.7    0.1

3.|-- 10.8.28.57                  0.0%   10   17.5   2.0   0.2   17.5   5.5

4.|-- 10.8.40.217                 0.0%   10   0.3    4.0   0.3   37.2   11.6

5.|-- 78.152.60.48                0.0%   10   0.4    0.4   0.4   0.5    0.0
   
6.|-- et-0-0-65.cr5-nyc2.ip4.gt   0.0%   10   8.9    9.5   8.4   15.7   2.2

7.|-- as15169.cr5-nyc2.ip4.gtt.   0.0%   10   8.6    8.6   8.5   8.6    0.0

8.|-- 108.170.248.65              0.0%   10   18.3   18.3  18.3  18.4   0.0

9.|-- 108.170.233.185             0.0%   10   18.4   18.4  18.3  18.4   0.0

10.|-- google-public-dns-a.googl  0.0%   10   10.3   10.3  10.2  10.3   0.0

As you can see the data is checked 10 times and statistics are provided with the best, worst and the average times.

mtr also provides packet loss information (similar to how ping works). A couple of additional options for mtr include

-c : Sets the number of iterations. Default is 10. For 20 iterations

$ mtr -r 8.8.8.8 -c 20

-i : Sets the number of secons per iteration. Usually, one iteration per second, but to identify how the system performs in a state of congestion, you can give a value < 1 to run more iterations per second. The following results in 5 iterations per second

$ mtr -r 8.8.8.8 -c 20 -i 0.2

In many cases, routing issues are temporary. Networks are resilient and usually alternative routes are found. However, these tools are useful to identify how data is sent and received from/to your server from various geographical points.