The mystery of the disappearing default routes
I was last week at a customer that had a strange problem: While installing the system the default route was disappearing. Those default routes are stored at the same place where all other persistent routes are stored and not at /etc/defaultrouterin Solaris 11 Just by setting /etc/defaultrouter they were able to set it. It took me a moment to find out was happening.
They installed the system via a 1 GBit/s interface and were moving afterwards to a group of IPMPed 10 Gbit/s interface for production purposes. After the move the default route was missing and they were setting it manually.
# ping c0t0d0s0.org
c0t0d0s0.org is aliveOkay, i’m now deleting the IP interface and renaming the data link:
# ipadm delete-ip net0
# dladm rename-link net0 production0Or use a different interface, that was the thing the customer did. Now configure an IP-address on this ip interface
# ipadm create-ip production0
# ipadm create-addr -T static -a 192.168.1.250/24 production/0Now I try to ping.
# ping c0t0d0s0.org
ping: sendto No route to hostOkay, it seems the default route isn’t working because i was able to ping systems in the same subnet. Let’s reboot the system in order to go to the standard default route setting procedure and
# rebootOkay, a short look into routing table.
$ netstat -nr
Routing Table: IPv4
Destination Gateway Flags Ref Use Interface
-------------------- -------------------- ----- ----- ---------- ---------
127.0.0.1 127.0.0.1 UH 2 48 lo0
192.168.1.0 192.168.1.250 U 3 11 production0 And so the result of the next command is inevitable … no route to host because there is indeed no route to host.
jmoekamp@solaris:~$ ping c0t0d0s0.org
ping: sendto No route to hostSo, when you look into the method script of svc:/network/routing-setup:default(the service responsible for setting at /lib/svc/method/net-routing-setup, you will find the following part at the end.
# Read /etc/inet/static_routes and add each route.
#
if [ -f /etc/inet/static_routes ]; then
echo "Adding persistent routes:"
/usr/bin/egrep -v "^(#|$)" /etc/inet/static_routes | while read line; do
/usr/sbin/route add $line
done
fiSo the persistent routes are read from the /etc/inet/static_routes file. Let’s have a look into it:
# cat /etc/inet/static_routes
# File generated by route(1M) - do not edit.
default 192.168.1.1 -ifp net0So let’s do the same as the script.
# route add default 192.168.1.1 -ifp net0
route: net0: no such interfaceSo this explains why the route is missing. In this case it’s the -ifp part. The IP interface simply doesn’t exists.
But where does this part originate from? It is set in the /lib/svc/method/net-install
if [ "$net_install_route" != "" ]; then
details="$details $net_install_route -ifp $ifp"
[...]
cmd="$ROUTE -R $rootdir -p add $details"
$cmd
[...]However there is a good question, why there is the the -ifp net0. Explicitly naming the interface is meant to side step several issues that could arise when having more than one interface in the system. And the behaviour of the system is perfectly reasonable as soon as you know that the default route as configured by the installation process isn’t in /etc/defaultrouter but /etc/inet/static_routes instead and is extended by the interface.
The next question was: Why did the /etc/defaultrouter worked. This behaves the way you know it.
In the method script /lib/svc/method/net-routing-setup you will find the following sequence of commands.
defrouters=`/usr/bin/grep -v \^\# /etc/defaultrouter | \
/usr/bin/awk '{print $1}'`
[...]
for router in $defrouters; do
route_added=`/usr/sbin/route -n add default \
-gateway $router`
res=$?
set -- $route_added
[ $res -ne 0 -a "$5" = "$route_IP:" ] && do_delete=no
doneSo this way to set the default route doesn’t set the default route with -ifp, so it works after a renaming in the same way. In the case of the customer, the default route set by the installation failed because of the -ifp with a non existing interface name, however before this the script using the /etc/defaultrouter already set a valid default route.
The solution to this? You could directly edit the file and change the interface name (or just deleting the -ifp net0 part. However there is a “do not edit marker”. Let us at least pretend for a second that we read such comments. The correct way is:
# route -p delete default 192.168.1.1
delete net default: gateway 192.168.1.1
delete persistent net default: gateway 192.168.1.1
# route -p add default 192.168.1.1 -ifp ipmp0
add net default: gateway 192.168.1.1
add persistent net default: gateway 192.168.1.1
# ping c0t0d0s0.org
c0t0d0s0.org is aliveHowever with IPMP everything is different
However for the most common case (i would say 90%) of an interface name change, it’s different. When you use IPMP the routes using the member interface are automatically translated to routes for the ipmp interface. The usual check if we can reach systems outside of our network.
# ping c0t0d0s0.org
c0t0d0s0.org is aliveOkay, a route is on net0 as the netstat output shows.
#netstat -nr
Routing Table: IPv4
Destination Gateway Flags Ref Use Interface
-------------------- -------------------- ----- ----- ---------- ---------
default 192.168.1.1 UG 2 1 net0
127.0.0.1 127.0.0.1 UH 2 48 lo0
192.168.1.0 192.168.1.250 U 3 11 net0
[...]I just delete the old interface and with it the configured addresses and create an IPMP group with just a single interface in it.
# ipadm delete-ip net0
# ipadm create-ip net0
# ipadm create-ipmp -i net0 ipmp0Okay, now i will assign the old address to it.
# ipadm create-addr -T static -a 192.168.1.251/24 ipmp0/v4Let’s check the routing table.
root@solaris:/home/jmoekamp# netstat -nr
Routing Table: IPv4
Destination Gateway Flags Ref Use Interface
-------------------- -------------------- ----- ----- ---------- ---------
127.0.0.1 127.0.0.1 UH 2 130 lo0
192.168.1.0 192.168.1.251 U 2 0 ipmp0
[...] Okay, route is missing. I will just restart the route configuration.
# svcadm restart routing-setupYet another look to the routing table.
# netstat -nr
Routing Table: IPv4
Destination Gateway Flags Ref Use Interface
-------------------- -------------------- ----- ----- ---------- ---------
default 192.168.1.1 UG 1 0 ipmp0
127.0.0.1 127.0.0.1 UH 2 154 lo0
192.168.1.0 192.168.1.251 U 4 54 ipmp0
[...]Et voila, there is a default gateway via ipmp0. However in the file with the persistent static routes, the route is still stored as a route for net0:
root@solaris:/home/jmoekamp# cat /etc/inet/static_routes
# File generated by route(1M) - do not edit.
default 192.168.1.1 -ifp net0The system automatically converts the route for net0 to a route for ipmp0. You can easily do it, as it’s a somewhat safe assumption, that a already existing route for interface would be valid as well for an ipmp group with this interface.
Okay, let’s just check if this persists a boot, so i’m rebooting the system.
jmoekamp@solaris:~$ netstat -nr
Routing Table: IPv4
Destination Gateway Flags Ref Use Interface
-------------------- -------------------- ----- ----- ---------- ---------
default 192.168.1.1 UG 2 15 ipmp0
127.0.0.1 127.0.0.1 UH 2 34 lo0
192.168.1.0 192.168.1.251 U 3 12 ipmp0
[...]
jmoekamp@solaris:~$ ping c0t0d0s0.org
c0t0d0s0.org is alive
jmoekamp@solaris:~$So it’s the same after aboot