Venus: Semi-Manual FreeBSD 11-CURRENT AMD64 ZFS+UEFI Installation

In this post I’ll be describing how to do a semi-manual installation of a FreeBSD 11 ZFS system with UEFI boot. Big thanks to Ganael Laplanche for this mailing list entry, as it was of great help. Some things have changed since then which makes the process a little simpler, and that’s why I’m writing this. :) I’ll also include some steps I consider best practices.

The steps outlined below are generalized from how I installed FreeBSD on my dev box named Venus.

As I’m writing this, the latest FreeBSD 11 snapshot is of r294912 (2016-01-27), and does not yet support automatic installation to ZFS on UEFI systems. I’m using this snapshot for installing the system.

Start the installer normally, and go through the steps. When you get to the part where it asks whether you want to install to UFS, ZFS, etc, chose to open a shell.

Create the partition scheme for each drive you will be using in your root zpool, and make sure to use unique labels. Make sure to replace ‘ada0’ with whatever is appropriate for you.
gpart create -s gpt ada0
gpart add -t efi -s 800k ada0
gpart add -t freebsd-zfs -a 1m -s 55g -l YourLabel ada0

I aligned the freebsd-zfs partition to 1M to ensure it’s 4k aligned, and to leave room for boot loader changes. I specified a 55GB partition because my SATADOM’s are 64GB, and I want to leave some free space in case I need to replace one of them with another which isn’t the exact same size, and because I want to leave some room for other things such as a future log, cache or swap partition.

Create the zpool and add datasets, then exit the shell. All datasets within sys/ROOT/default are optional.
zpool create -m none -o altroot=/mnt -O atime=off -O checksum=fletcher4 -O compress=lz4 sys gpt/YourLabel
zpool set bootfs=sys/ROOT/default sys
zfs create -p sys/ROOT/default/var
zfs create -o compress=gzip-9 -o setuid=off sys/ROOT/default/var/log
zfs create -o compress=gzip-9 -o setuid=off sys/ROOT/default/var/tmp
zfs create sys/ROOT/default/usr
zfs create -o compress=gzip-9 sys/ROOT/default/usr/src
zfs create sys/ROOT/default/usr/obj
zfs create sys/ROOT/default/usr/local
zfs create sys/data
zfs create -o mountpoint=/usr/home -o setuid=off sys/data/homedirs
zfs mount -a

Now the installer should continue doing its thing. Do what you’d normally do, but when it asks if you want to open a shell into the new environment, say yes.

Execute this commands to ensure ZFS mounts all datasets on boot:
echo 'zfs_enable="YES"' >> /etc/rc.conf

Configure the (U)EFI partitions by doing the following for each drive that is a member of the ‘sys’ zpool: (remember to replace ‘ada0’ with whatever is appropriate for you)
mkdir /mnt/ada0
newfs_msdos ada0p1
mount -t msdosfs /dev/ada0p1 /mnt/ada0
mkdir -p /mnt/ada0/efi/boot
cp /boot/boot1.efi /mnt/ada0/efi/boot/BOOTx64.efi
mkdir -p /mnt/ada0/boot
cat > /mnt/ada0/boot/loader.rc << EOF
set currdev=zfs:sys/ROOT/default:
load boot/kernel/kernel
load boot/kernel/zfs.ko

At this time you can double check you have the expected file hierarchy in /mnt/ada0:

(cd /mnt/ada0 && find .)

Should output:

Now, if you had more than one drive, you can just copy the contents of /mnt/ada0 to the appropriate mountpoints. cp -R /mnt/ada0/ /mnt/ada1/

Remember to unmount the EFI partitions, then exit the shell and reboot into the new system. :)

Once you’re in the new system, you should create a read-only ZFS dataset for /var/empty.

PS: Similar to how you need to re-apply bootcode when upgrading zpool version, you should probably re-copy /boot/loader.efi to the EFI partition as ./efi/boot/BOOTx64.efi. I am not sure if this is strictly necessary… But it shouldn’t hurt. :) I’ll update this paragraph when I get a confirmation one way or the other.

Introduction: Venus, the FreeBSD dev box

(Also known as: The Mini-ITX quarter-depth chassis that could fit a Micro-ATX mainboard)

I’ll be using this system for my FreeBSD hacking, but this post focus on the system hardware.

The story of this system started when I ordered my Super Micro quarter-depth (SC505-203B) Atom-based firewall named Kuiper from Nextron. The chassis specifications state it’s 24.9cm deep and 43.7cm wide (9.8″ and 17.2″ respectively), and would only fit mini-ITX boards.

As I was also interested in a Xeon E3v5 virtualization server, and would prefer it to be quarter-depth as well, I was a little disappointed that Super Micro didn’t have any mini-ITX mainboards for that platform. Nextron helpfully suggested that they could check if the Super Micro X11SSL-F mainboard (micro-ATX) would fit when building my firewall, as they had it on hand. It has the dimensions 24.4cm by 24.4 cm (9.6″ by 9.6″). I was not expecting it to fit as the depth of the mainboard was a mere 5 mm (0.2″) less than the chassis.

A few days later, when the firewall was built, they reported back: It fits! But they would have to sacrifice one of two 2×2.5″ drive bays. It was also a very, very snug fit, as can be seen in the image below. Excellent! I only need two data drives in that system anyway, and COULD use them as root drives if necessary.

Snug Fit 2

Snug fit! This is NOT the I/O side!

I was happy. Now, considering this chassis was designed for Atom systems, the PSU would probably not be capable of powering a 80W CPU, for not to mention the potential cooling trouble. Nextron suggested getting a 45W CPU, but I decided to pay the premium of getting a low-powered CPU, the Xeon E3-1240L, with a TDP of only 25W.

System Parts
Chassis SuperChassis CSE-505-203B
Mainboard Super Micro X11SSL-F
CPU Xeon E3-1240L
RAM 32GB: 2x 16GB DDR4 2133MHz ECC Unbuffered DIMM
HDD 2x Seagate Laptop Thin SSHD HDD/SSD Hybrid – 500GB SATA3 5400RPM 2.5″
SSD 2x Supermicro SATADOM 64GB MLC – Vertical (added later)

Picture Gallery

Please note the pictures above were taken at different points throughout my process of modifying the system. The final setup (for now) has two SATADOMs and two SSHD’s, and is shown in this posts featured image, and is the last entry in the gallery above.

I should probably also mention that the chassis ‘curvature’/apparent bending seen in some of the photos is a trick of the lens.