A Tale of Two Linux Routers

By default, administrators just define a single, default router (which is eth0). So for a Linux system with two network interface cards —  eth0 and eth1, and if we receive traffic (i.e., ICMP pings) on eth1, the return traffic will go out on eth0.

In short, this post will explain how to ensure traffic going into eth1 goes out only on eth1, as well as enforce all traffic going into eth0 goes out only on eth0.

Assuming we have the following network setup:

• eth0 - 10.10.1.10 netmask 255.255.255.0
• eth0's gateway is: 10.10.1.254
• eth1 - 192.168.7.7 netmask 255.255.255.0
• eth1's gateway is: 192.168.7.1

First, we need to make sure the Linux kernel has support for “policy routing” enabled.

During the kernel compilation process, we need to:

cd /usr/src/linux
make menuconfig
Select "Networking --->"
Select "Networking options --->"
[*] TCP/IP networking
[*] IP: advanced router
Choose IP: FIB lookup algorithm (FIB_HASH)
[*] IP: policy routing
[*] IP: use netfilter MARK value as routing key

Next, we need to download, compile, and install the iproute2 utilities. (Most Linux distributions have binary packages for this utility.) Once installed, typing ip route show should show the system’s routing table. 

To check the system’s initial route configuration:

# netstat -anr
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
192.168.7.0  0.0.0.0  255.255.255.0 U 0 0 0 eth1
10.10.1.0  0.0.0.0  255.255.255.0 U 0 0 0 eth0
0.0.0.0  192.168.7.1  0.0.0.0 UG 0 0 0 eth1

So, basically, the system is using eth1 as the default route. If anyone pings 192.168.7.7, then the response packets will properly go out eth1 to the upstream gateway of 192.168.7.1. But what about pinging 10.10.1.10? The incoming ICMP packets will properly arrive on eth0, but the outgoing response packets will be sent out via eth1! That is not good.

To fix this issue, we need to create a new policy routing table entry within the /etc/iproute2/rt_tables. We call it table #1, named “admin” (for routing administrative traffic onto eth0).

# echo "1 admin" >> /etc/iproute2/rt_tables

Then, we are going to add a few new entries within this “admin” table. Specifically, we provide information about eth0‘s local /24 subnet, along with eth0‘s default gateway.

ip route add 10.10.1.0/24 dev eth0 src 10.10.1.10 table admin
ip route add default via 10.10.1.254 dev eth0 table admin

At this point, we have created a new, isolated routing table named “admin” that is not used by the OS just yet. Because we still need to create a rule referencing how the OS should use this table. For starters, type ip rule show to see your current policy routing ruleset. Here’s what an empty ruleset looks like:

0: from all lookup local
32766: from all lookup main
32767: from all lookup default

Without going into all the boring details, each rule entry is evaluated in ascending order. The main gist is that the normal main routing table appears as entry 32766 in this list. (This would be the normal route table we see when we type netstat -anr.)

We are now going to create two new rule entries, that will be evaluated before the main rule entry.

ip rule add from 10.10.1.10/32 table admin
ip rule add to 10.10.1.10/32 table admin

Typing ip rule show now shows the following policy routing rulesets:

0: from all lookup local
32764: from all to 10.10.1.10 lookup admin
32765: from 10.10.1.10 lookup admin
32766: from all lookup main
32767: from all lookup default

Rule 32764 specifies that for all traffic going to eth0‘s IP, make sure to use the “admin” routing table, instead of the “main” one. Likewise, rule 32765 indicates that for all traffic originating from eth0‘s IP, make sure to use the “admin” routing table as well. For all other packets, use the “main” routing table. In order to commit these changes, it’s a good idea to type ip route flush cache.

So the system should now be able to properly route traffic to these two different default gateways. 

References:
[1] http://www.policyrouting.org
[2] http://www.linuxhorizon.ro/iproute2.html

Update: Here are some additional resources, that I have found useful.
http://lartc.org/howto/lartc.rpdb.multiple-links.html
http://linux-ip.net/html/routing-tables.html