Last week I had no problem ssh’ing to one of the production servers from my main development machine. This week and for no apparent reason a ping results in “Destination Host Unreachable”, ssh’ing to the server results in “No route to host”. Certainly a networking issue but where do you start?
My networking skill are limited, I’ve always found networking a daunting, does this have to be the case? There are so many things that could go wrong; routing issue, closed ports, firewall rules etc etc. Unless you’re in the know it’s really difficult to problem solve. As with everything, using the right tool for the job makes life a little easier!
Let’s recap:
- I can no longer ssh to a server I could recently
- Pinging the server gives Destination Host Unreachable
Where to start:
I want to see where my traffic is being routed, for this I’m going to use tracepath
%> tracepath scottfreeit.dmz1?: [LOCALHOST] pmtu 1500
1: 192.168.48.1 3071.104ms
Resume: pmtu 1500
I’d expect to see some more entries that this, also the last entry should be the IP address of my scottfreeit.dmz server (192.168.0.144).
There’s something fishy about the results of the tracepath. The IP address 192.168.48.1 appears to be on my local network (192.168) but the rest of the IP address I don’t recognise (48.1). I don’t recall having anything configured to use the 48 range. Wherever this is coming from I’m confident this is the source of my problem.
After running an ifconfig command to display all configured network interfaces on I see an interface matching the 192.168.48.1
br-1b80ed20c2c3 Link encap:Ethernet HWaddr 02:42:1c:0e:bc:a0
inet addr:192.168.48.1 Bcast:172.19.255.255 Mask:255.255.0.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Something else I didn’t recognise; I had an interface named br-1b80ed20c2c3!? After a little head scratching and trying to remember what has changed between successfully connecting to my remote server and now the penny dropped. I had recently been working with docker and creating various virtual networks to allow various containers to communicate with each other, lets take a look at these virtual networks
%> docker network ls
NETWORK ID NAME DRIVER SCOPE
3c56404c1b88 bridge bridge local
1b80ed20c2c3 grafana-bridge bridge local
There’s the culprit Grafana-bridge. To remove this interface I used the ip command, this command shows/manipulates routing and devices
%> sudo ip link delete 1b80ed20c2c3
The interface no longer appeared in the ifconfig results and a tracepath command returned the following
%> tracepath scottfreeit.dmz1?: [LOCALHOST] pmtu 1500
1: 192.168.0.144 4.896ms reached
1: 192.168.0.144 3.395ms reached
Resume: pmtu 1350 hops 1 back 1
Great, no sign of the problematic IP address and indeed replaced with the correct IP for my server! The last thing to try was to try connecting to the scottfreeit.dmz server via ssh, bingo, it worked!
Onto the next problem!