Venus: Semi-Manual FreeBSD 11-CURRENT AMD64 ZFS+UEFI Installation

In this post I’ll be describing how to do a semi-manual installation of a FreeBSD 11 ZFS system with UEFI boot. Big thanks to Ganael Laplanche for this mailing list entry, as it was of great help. Some things have changed since then which makes the process a little simpler, and that’s why I’m writing this. :) I’ll also include some steps I consider best practices.

The steps outlined below are generalized from how I installed FreeBSD on my dev box named Venus.

As I’m writing this, the latest FreeBSD 11 snapshot is of r294912 (2016-01-27), and does not yet support automatic installation to ZFS on UEFI systems. I’m using this snapshot for installing the system.

Start the installer normally, and go through the steps. When you get to the part where it asks whether you want to install to UFS, ZFS, etc, chose to open a shell.

Create the partition scheme for each drive you will be using in your root zpool, and make sure to use unique labels. Make sure to replace ‘ada0’ with whatever is appropriate for you.
gpart create -s gpt ada0
gpart add -t efi -s 800k ada0
gpart add -t freebsd-zfs -a 1m -s 55g -l YourLabel ada0

I aligned the freebsd-zfs partition to 1M to ensure it’s 4k aligned, and to leave room for boot loader changes. I specified a 55GB partition because my SATADOM’s are 64GB, and I want to leave some free space in case I need to replace one of them with another which isn’t the exact same size, and because I want to leave some room for other things such as a future log, cache or swap partition.

Create the zpool and add datasets, then exit the shell. All datasets within sys/ROOT/default are optional.
zpool create -m none -o altroot=/mnt -O atime=off -O checksum=fletcher4 -O compress=lz4 sys gpt/YourLabel
zpool set bootfs=sys/ROOT/default sys
zfs create -p sys/ROOT/default/var
zfs create -o compress=gzip-9 -o setuid=off sys/ROOT/default/var/log
zfs create -o compress=gzip-9 -o setuid=off sys/ROOT/default/var/tmp
zfs create sys/ROOT/default/usr
zfs create -o compress=gzip-9 sys/ROOT/default/usr/src
zfs create sys/ROOT/default/usr/obj
zfs create sys/ROOT/default/usr/local
zfs create sys/data
zfs create -o mountpoint=/usr/home -o setuid=off sys/data/homedirs
zfs mount -a
exit

Now the installer should continue doing its thing. Do what you’d normally do, but when it asks if you want to open a shell into the new environment, say yes.

Execute this commands to ensure ZFS mounts all datasets on boot:
echo 'zfs_enable="YES"' >> /etc/rc.conf

Configure the (U)EFI partitions by doing the following for each drive that is a member of the ‘sys’ zpool: (remember to replace ‘ada0’ with whatever is appropriate for you)
mkdir /mnt/ada0
newfs_msdos ada0p1
mount -t msdosfs /dev/ada0p1 /mnt/ada0
mkdir -p /mnt/ada0/efi/boot
cp /boot/boot1.efi /mnt/ada0/efi/boot/BOOTx64.efi
mkdir -p /mnt/ada0/boot
cat > /mnt/ada0/boot/loader.rc << EOF
unload
set currdev=zfs:sys/ROOT/default:
load boot/kernel/kernel
load boot/kernel/zfs.ko
autoboot
EOF

At this time you can double check you have the expected file hierarchy in /mnt/ada0:

(cd /mnt/ada0 && find .)

Should output:
.
./efi
./efi/boot
./efi/boot/BOOTx64.efi
./boot
./boot/loader.rc

Now, if you had more than one drive, you can just copy the contents of /mnt/ada0 to the appropriate mountpoints. cp -R /mnt/ada0/ /mnt/ada1/

Remember to unmount the EFI partitions, then exit the shell and reboot into the new system. :)

Once you’re in the new system, you should create a read-only ZFS dataset for /var/empty.

PS: Similar to how you need to re-apply bootcode when upgrading zpool version, you should probably re-copy /boot/loader.efi to the EFI partition as ./efi/boot/BOOTx64.efi. I am not sure if this is strictly necessary… But it shouldn’t hurt. :) I’ll update this paragraph when I get a confirmation one way or the other.

A farewell to the old, and a hello to the new

It’s late August, which means it’s currently late summer, and soon fall, where I live. It’s time to decide on a winter project. Previously, I’ve had coding sprees on various things related to the MMORPG Anarchy Online. Some examples, in no particular order: AOItems (an items database), Towerwar Tracker (mobile) (shows Player versus Player land control fights), PlanetMap Viewer, and more. Although I have to admit some of those stretched into year-around and even multi-year projects. :) In idle moments, I’ve taken stabs at improving things in the FreeBSD land as best as I can, by writing some guides and giving a helping hand on various social media, forums and IRC.

Anarchy Online has been the main focus point of many of my IT-related hobby projects. And much of my home infrastructure, powered by FreeBSD, is the way it is now because it had to be that way to support and run those projects. I started playing the game in December 2004, and stopped playing it early 2009. I have played it a week here or there when large patches have hit, but for the most part, just logged in to check if my AO-related tools were still working, or in some other way related to improving those tools. It’s now August 2015, and it’s six years since I stopped playing that game. I’ve been making tools for the game for more than ten years, and it has been fun. It has been an incredible learning experience. I’ve done many crazy things, such as:

  • Making a multi-process map compiler using PHP and syncing state on disk (4 processes was about 50% faster than one)
  • Porting said compiler to C#.Net, heavily optimized with threads. About 40x faster than single-process PHP version :)
  • Creating a LARGE toolchain (20 libraries/applications) for extracting item info from the game client and making sense out of it, storing it in a DB and displaying it on a website using PHP. Toolchain parses through 14 years of patch history in about 15 minutes. (similar tools were said to be taking weeks to do the same). This toolchain also verifies integrity, and automatically creates reports on human-induced mistakes in the data, for easy and detailed report submission to the games developers.
  • For the planetmap viewer, using a hooking library to run my own code in the context of the game client, effectively using the game client as an API.
  • Much more :)

If it wasn’t for this game, its awesome community, and all the related projects I’ve worked on, I wouldn’t be where I am today. I wouldn’t have known C#.NET as well as I do, if at all. There are many friends I would never have made otherwise, all over the world. If you ever read this, you should know who you are. :)

All that being said, 10 years is a long time. It’s a third of my life on this planet. It was fun, it was quite the experience. It really was. But now it’s time to change focus. I will maintain the existing projects for the foreseeable future, until a suitable successor can be found. No new features are likely to arrive. All the tools except those related to AOItems.com are open source, so anyone who’s up to the task can fork and improve them. I’ll keep AOItems updated for the foreseeable future. If I ever stop maintaining it, I will make the tools available to the community, so that others can pick up the torch where I left it. People can do amazing things if you let them. :)

To the whole Anarchy Online community, who know me as “Demoder”: Thank you for being truly awesome, and inspiring me to do ‘crazy’ things. I’ll try not to be a stranger!

The New

I plan on learning to play the (musical) keyboard. Exercise more. Be more out-going. Maybe quit a bad habit or two. All of those things that people write on their blogs, and sometimes follow up on. And often don’t. But there’s more.

I was introduced to the world of FreeBSD and Linux back in 2001 or so. The story is long and for another blog post, but the point is this: FreeBSD has been with me in varying degrees for nearly 15 years, or half my life, and I’m now at a point in life where I feel like contributing more than I have in the past. I started earlier this year, submitting a PR and a patch for adding LibXo support to iscsictl(8), and then proceeding with a thorough technical review of the book “FreeBSD Mastery: ZFS” by Michael W. Lucas and Allan Jude. And I feel these are the kinds of things I want to do with the large chunk of my spare time which is labeled “geeky things”. Make contributions to something, hopefully making a positive change for other people in the process.

I’ve been using jails for a long time. For the Linux folks out there, think containers. For the Solaris people out there, think zones. I love how easy it is to manage large amounts of data with ZFS, and how trouble-free it is to share this data with the jail environments, with (nearly) no overhead.

The past couple of years, some awesome people have been implementing a native hypervisor to FreeBSD called “Bhyve”. I love it. I really do. It lets me do things that jails wouldn’t. It will eventually let me retire my VMWare ESXi server. There are a few things which are annoying with it though, but most of those are being worked on by very skilled people. The one feature I miss the most,  is an easy and cheap way to leverage ZFS with no performance penalty. Think of jails with nullfs, or having a ZFS dataset delegated to a jail. That’s super awesome.

My primary use case would be to set up a virtualized file server, leveraging ZFS on the host without ZFS on ZFS or similar overhead, and avoiding the growing complexity of things like NFS configurations. As such, my new winter project is FreeBSD-related. It involves Bhyve (hypervisor), file systems, host/vm communication, and simplifying administration. It involves learning C properly, virtio, FUSE, and FreeBSD kernel internals. Its name is Tunnel File System (https://tunnelfs.io/). The goal is to share files between host and guest like you do between host and jails using nullfs. It aims to be simple to configure/use by the system administrator. It aims to be predictable. It aims to make your life simpler. You can read about it in more detail on the project’s site.

I have a pretty good idea (or so i think right now!) on how to implement it. I know what I need to learn. I know I have a LOT to learn. But that’s okay. I LOVE learning. Almost anything fun in life involves learning something new. So I’ll get started on that. And I’ll probably find out that I can’t do these things the way I wanted, but that’s okay too. It’s a learning experience, and I will get to the finish line eventually. :)

PS: If you haven’t read FreeBSD Mastery: ZFS, and use or plan to use ZFS, you should go read it. Even if you’re not using FreeBSD. If you don’t own it, you should buy it. It’s awesome, and it will look good on your book shelf.

Converting a FreeBSD MySQL server to jail host with MySQL in jail

I have a FreeBSD 10.0 server which currently only runs Percona MySQL server 5.6 backed by ZFS. The SQL server doesn’t have a high enough load to justify dedicated hardware, but I also don’t want to run it as a virtual machine as I want to use local ZFS storage, and because of virtualization overhead. The server is dual-homed (DMZ and LAN).

The solution is to convert the server into a jail host, and run MySQL inside a jail. The overhead should be minimal to non-existing as I won’t be using VNET.
Continue reading

FreeBSD jail server with ZFS clone and jail.conf

I’ll be using FreeBSD 10.0 AMD64 with root on ZFS, but you can follow these instructions as long as you have a ZFS pool on the system. It is assumed that the system is already installed and basic configuration is complete.

It should be noted that the benefit from using ZFS clones will more or less vanish if you do a major ‘world’ upgrade on the jail, for example upgrading from FreeBSD 9.2 to FreeBSD 10.0. This won’t be a problem for my setup as I’ll eventually get around to configuring sysutils/py-salt to automatically deploy my jails, and I’ll post about it when I do.
Continue reading

FreeBSD package builder with Poudriere

I’m using FreeBSD 10.0-RELEASE on my file server, which will double as my package builder. I’d prefer to run Poudriere inside a jail  so that all its binaries and configs are confined there, but this is not a supported configuration, and Poudriere requires so many permissions the security benefits would be minimal, and it still encounters trouble.

This shouldn’t be a problem though, as Poudriere won’t expose any services, and the packages will be published by a jail utilizing read-only nullfs mounts.

Continue reading

FreeNAS ZFS Benchmarks: 1 Data Drive

This benchmark have three pool configurations: Single drive, a 2-way mirror, and a 3-way mirror. Please see the previous posts on testing methology and hardware specifications, if you haven’t already.

I expect the single drive to have the best write performance, followed by the 2-way then 3-way mirror. I also expect the mirrors to have better read performance, as there’s more drives to read from. I also expect there to be a noticeable performance penalty for record sizes smaller than ZFS’s configured recordsize of 128k. I expect error margins of +/- 10%. The random reads/writes should get almost linearly better performance with the larger record sizes.

Continue reading

Benchmarking HOMER+FreeNAS: Introduction and methology

There are many claims about the performance of ZFS. As I’m about to replace a nearly 4 years old file server, I decided to run thorough tests to see how the different pool configurations available to my 12-drive system will actually perform. I’ve tested all the vdev types (single drive, mirror, raidz[1-3]), and stripes thereof. For more information about the involved hardware, please see the full specifications of the server used for these benchmarks.

I’ve used FreeNAS 9.2-RC2 (x64), and IOZone v3.420 (compiled for 64-bit mode, build: freebsd) for these benchmarks. I disabled SWAP in the FreeNAS settings prior to creating any test pool, in an effort to prevent arbitary i/o from skewing results. I also disabled the ‘atime’ property on the test pool, to reduce unnecessary I/O. The benchmarks were run inside a 64-bit portjail, nullfs-mounting the test pool to /mnt inside the jail. SSHD is the only started non-default service. The jail was running on a ZFS pool consisting of a SSD mirror.

Continue reading