FreeBSD jail host with multiple local networks

My jail host is running FreeBSD 10.0-RELEASE and is directly connected to two local networks. One is my LAN, and the other is a DMZ for various internet-facing services. I don’t want my DMZ jails to be able to send network traffic directly to my LAN, and I need to set a default route for a jail depending on which network its IP-address resides for them to communicate outside of their local subnet.

To solve this, I’m going to use multiple routing tables, also known as FIB, which are manipulated with the setfib utility. I know I could have used the experimental virtual network stack (VNET), which is awesome, but I opt not to as it still has some problems with stability and memory leaks. EDIT: It seems that jails are able to use the ‘setfib’ command as well, so a firewall might be necessary to disallow communication between certain jails and destinations.

I defined the following tunables in /boot/loader.conf followed by a reboot:

# file: /boot/loader.conf
net.fibs=4
net.add_addr_allfibs=0

The first tunable, net.fibs, defines how many routing tables I want to have. I only need two, but I figure it’s good to have some room to grow as the host has to reboot for that change to take effect.

The second tunable, net.add_addr_allfibs, defines whether to add default route and interface routing entries to all routing tables. Setting this to 0 (disabled) should solve the problem with cross-talk and default route, as only routing table 0 (default; the one the host uses) will be populated with default routes and interface-local addresses during start-up, or during further interface manipulation.

However, I now have one default routing table which works as expected, and three empty routing tables. I’ll have to manually populate those routing tables. This is done by placing a few commands in /etc/rc.local (please let me know if there is a cleaner alternative!):

#!/bin/sh
# file: /etc/rc.local
##
# Routing Tables
##
# 0: Default
# 1: DMZ
# 2:
# 3:
#
##
# Set up the DMZ[1] routing table
##
# Interface route(s)
setfib 1 route add -net 10.5.2.0/24 -iface V_DMZ
# Default route
setfib 1 route add default 10.5.2.1

Remember to manually execute the commands to update the routing tables (or reboot). Now everything is ready for the jails to use this new routing table.

All that remains now is to configure the jail to use the alternate routing table. This is done by manipulating the exec.fib jail parameter. Example using /etc/jail.conf:

# file: /etc/jail.conf

# Insert your defaults here

# My DMZ jail:
pkg {
  # Set to the DMZ routing table
  exec.fib=1;
  interface = "V_DMZ";
  ip4.addr = 10.5.2.X;
  # Debugging purposes
  allow.raw_sockets = 1;
}

I restarted the jail, and it now behaved as I wanted it to. :)

EDIT: If you execute commands inside the jail using ‘jexec’, the current process’ routing table will be set to the default routing table. To properly execute commands inside the jail, you’ll either have to SSH in, or manually adjust the fib with ‘setfib’ when using jexec.