FreeBSD jail host with multiple local networks

My jail host is running FreeBSD 10.0-RELEASE and is directly connected to two local networks. One is my LAN, and the other is a DMZ for various internet-facing services. I don’t want my DMZ jails to be able to send network traffic directly to my LAN, and I need to set a default route for a jail depending on which network its IP-address resides for them to communicate outside of their local subnet.

To solve this, I’m going to use multiple routing tables, also known as FIB, which are manipulated with the setfib utility. I know I could have used the experimental virtual network stack (VNET), which is awesome, but I opt not to as it still has some problems with stability and memory leaks. EDIT: It seems that jails are able to use the ‘setfib’ command as well, so a firewall might be necessary to disallow communication between certain jails and destinations.

I defined the following tunables in /boot/loader.conf followed by a reboot:

# file: /boot/loader.conf
net.fibs=4
net.add_addr_allfibs=0

The first tunable, net.fibs, defines how many routing tables I want to have. I only need two, but I figure it’s good to have some room to grow as the host has to reboot for that change to take effect.

The second tunable, net.add_addr_allfibs, defines whether to add default route and interface routing entries to all routing tables. Setting this to 0 (disabled) should solve the problem with cross-talk and default route, as only routing table 0 (default; the one the host uses) will be populated with default routes and interface-local addresses during start-up, or during further interface manipulation.

However, I now have one default routing table which works as expected, and three empty routing tables. I’ll have to manually populate those routing tables. This is done by placing a few commands in /etc/rc.local (please let me know if there is a cleaner alternative!):

#!/bin/sh
# file: /etc/rc.local
##
# Routing Tables
##
# 0: Default
# 1: DMZ
# 2:
# 3:
#
##
# Set up the DMZ[1] routing table
##
# Interface route(s)
setfib 1 route add -net 10.5.2.0/24 -iface V_DMZ
# Default route
setfib 1 route add default 10.5.2.1

Remember to manually execute the commands to update the routing tables (or reboot). Now everything is ready for the jails to use this new routing table.

All that remains now is to configure the jail to use the alternate routing table. This is done by manipulating the exec.fib jail parameter. Example using /etc/jail.conf:

# file: /etc/jail.conf

# Insert your defaults here

# My DMZ jail:
pkg {
  # Set to the DMZ routing table
  exec.fib=1;
  interface = "V_DMZ";
  ip4.addr = 10.5.2.X;
  # Debugging purposes
  allow.raw_sockets = 1;
}

I restarted the jail, and it now behaved as I wanted it to. :)

EDIT: If you execute commands inside the jail using ‘jexec’, the current process’ routing table will be set to the default routing table. To properly execute commands inside the jail, you’ll either have to SSH in, or manually adjust the fib with ‘setfib’ when using jexec.

5 thoughts on “FreeBSD jail host with multiple local networks

  1. You can also do this directly in rc.conf

    static_routes=”dmz_if dmz_gw”
    route_dmz_if=”-net 10.5.2.0/24 -iface V_DMZ -fib 1″
    route_dmz_gw=”default 10.15.2.1 -fib 1″

    Like

  2. Because of the introduction of SR-IOV, this approach to jail-networking has some really interesting perspectives with FreeBSD 11 and later. That way one could have a pile of virtual NICs and each jail being assigned a dedicated virtual NIC rather than having to do bridging and other stuff.

    Like

    • That’s true. Would need VNET to fully utilize it though, and there are still some quirks with that. It’s a lot better now than it was in the past, though!

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s