FreeBSD 9 file server: Wiggum version 2.0

I’ve recently upgraded Wiggum (my file server) from FreeBSD 8.0 to FreeBSD 9.0.  Since I had made some mistakes when originally setting up Wiggum two years ago, I went for a complete reinstall – and recreation of the zpools. This blog entry is a step-by-step guide for how I did the initial installation and setup.

Specifications

  • CPU: Intel Core i7 930 @ 2.8GHx (no change)
  • MainboardGigabyte GA-X58A-UD3R (no change)
  • PSUCooler Master Silent Pro M600 600W (no change)
  • RAM: Corsair XMS3 DDR3 1600MHz 12GB CL9 (no change)
  • Extra SATA: HighPoint RocketRAID 2320 8P SATA (no change)
  • NIC: 2x Intel Pro/1000CT PCIe Desktop Adapter (no change)
  • Storage:
    • 3x Intel® SSD 330 Series 120GB, which will be used for OS, swap, ZFS log and ZFS cache
    • 12x 1.5TB HDD (6x WDC GreenPower and 6x Seagate Barracuda 7200.11), which will be used for zpools.

Partitioning

System

The SSDs are partitioned with a boot sector, 8GB for OS, 1GB for swap, 2x 6GB for ZFS log, 10+80GB for ZFS cache. OS and swap are 3-way GEOM mirrors, while ZFS log is a 3-way ZFS mirror. Cache has no mirroring.
I booted the FreeBSD 9.0 AMD64 install CD (disk1), went through the installer as normal until asked about partitioning. Here, I chose to enter shell, and executed the following commands:

# Support for geom mirroring
kldload geom_mirror

### Start: For each system drive:
gpart create -s gpt da0
gpart add -t freebsd-boot -l Bay1.X-boot -s 64 da0
gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 da0
gpart add -a 4m -t freebsd-ufs -l Bay1.X-system -s 8G da0
gpart add -a 4m -t freebsd-swap -l Bay1.X-swap -s 1G da0

# log device should be half of system ram at most, according to http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide#Separate_Log_Devices
# log for backup pool
gpart add -a 4m -t freebsd-zfs -l Bay1.X-log1 -s 6G da0
# log for storage pool
gpart add -a 4m -t freebsd-zfs -l Bay1.X-log2 -s 6G da0

# Cache for backup pool
gpart add -a 4m -t freebsd-zfs -l Bay1.X-cache1 -s 10G da0
# Cache for storage pool
gpart add -a 4m -t freebsd-zfs -l Bay1.X-cache2 -s 80G da0
### END: For each system drive

# Partitions are now a-okay. Create geom mirrors!
gmirror label -v -b round-robin wiggum-system /dev/gpt/Bay1.1-system /dev/gpt/Bay1.2-system /dev/gpt/Bay1.3-system
gmirror label -v -b round-robin wiggum-swap /dev/gpt/Bay1.1-swap /dev/gpt/Bay1.2-swap /dev/gpt/Bay1.3-swap

# Format the system partition
newfs -U -t -S 4k /dev/mirror/wiggum-system
# Mount the system partition where the installer expects it
mount /dev/mirror/wiggum-system /mnt

# Create a fstab which will be installed by the installer
echo "/dev/mirror/wiggum-swap           none  swap   sw    0 0" > /tmp/bsdinstall_etc/fstab
echo "/dev/mirror/wiggum-system         /     ufs    rw      1 1" >> /tmp/bsdinstall_etc/fstab

# Resume installer by typing "exit"

After the installer is done, it will ask you if you want to make any final changes in a shell or reboot. Opt to use the shell. This shell will be chrooted into your new OS.
Execute the following command in the shell:

echo 'geom_mirror_load="YES"' >>/boot/loader.config

Data

After booting the system, I removed the partition tables on the data drives, then partitioned them as follows:

gpart create -s gpt adaX
gpart add -a 4k -t freebsd-zfs -l "BayX.Y adaX

and then, I created the pools:

gnop create -S 4k /dev/gpt/Bay*
zpool create backup raidz1 gpt/Bay2.2.nop gpt/Bay2.3.nop gpt/Bay3.*.nop log mirror gpt/Bay1.*-log1.nop cache gpt/Bay1.*-cache1
zpool create storage raidz2 gpt/Bay4.*.nop gpt/Bay5.*.nop log mirror gpt/Bay1.*-log2.nop cache gpt/Bay1.*-cache2 spare gpt/Bay2.1.nop

zpool export backup
zpool export storage
gnop destroy /dev/gpt/*.nop
zpool import -d /dev/gpt backup
zpool import -d /dev/gpt storage

And the end result (retrieved a week later):

zpool status
  pool: backup
 state: ONLINE
 scan: scrub repaired 0 in 0h10m with 0 errors on Wed Aug 29 01:10:02 2012
config:

        NAME                               STATE     READ WRITE CKSUM
        backup                             ONLINE       0     0     0
          raidz1-0                         ONLINE       0     0     0
            gpt/Bay2.2-Seagate-XXXXXXXX    ONLINE       0     0     0
            gpt/Bay2.3-Seagate-XXXXXXXX    ONLINE       0     0     0
            gpt/Bay3.1-WDCGP-XXXXXXXXXXXX  ONLINE       0     0     0
            gpt/Bay3.2-WDCGP-XXXXXXXXXXXX  ONLINE       0     0     0
            gpt/Bay3.3-WDCGP-XXXXXXXXXXXX  ONLINE       0     0     0
        logs
          mirror-1                         ONLINE       0     0     0
            gpt/Bay1.1-log1                ONLINE       0     0     0
            gpt/Bay1.2-log1                ONLINE       0     0     0
            gpt/Bay1.3-log1                ONLINE       0     0     0
        cache
          gpt/Bay1.1-cache1                ONLINE       0     0     0
          gpt/Bay1.2-cache1                ONLINE       0     0     0
          gpt/Bay1.3-cache1                ONLINE       0     0     0

errors: No known data errors

  pool: storage
 state: ONLINE
 scan: scrub repaired 32K in 3h52m with 0 errors on Sun Aug 26 04:29:51 2012
config:

        NAME                               STATE     READ WRITE CKSUM
        storage                            ONLINE       0     0     0
          raidz2-0                         ONLINE       0     0     0
            gpt/Bay4.1-Seagate-XXXXXXXX    ONLINE       0     0     0
            gpt/Bay4.2-Seagate-XXXXXXXX    ONLINE       0     0     0
            gpt/Bay4.3-Seagate-XXXXXXXX    ONLINE       0     0     0
            gpt/Bay5.1-WDCGP-XXXXXXXXXXXX  ONLINE       0     0     0
            gpt/Bay5.2-WDCGP-XXXXXXXXXXXX  ONLINE       0     0     0
            gpt/Bay5.3-WDCGP-XXXXXXXXXXXX  ONLINE       0     0     0
        logs
          mirror-1                         ONLINE       0     0     0
            gpt/Bay1.1-log2                ONLINE       0     0     0
            gpt/Bay1.2-log2                ONLINE       0     0     0
            gpt/Bay1.3-log2                ONLINE       0     0     0
        cache
          gpt/Bay1.1-cache2                ONLINE       0     0     0
          gpt/Bay1.2-cache2                ONLINE       0     0     0
          gpt/Bay1.3-cache2                ONLINE       0     0     0
        spares
          gpt/Bay2.1-Seagate-9VS44MX8      AVAIL

errors: No known data errors

gmirror status
                Name    Status  Components
mirror/wiggum-system  COMPLETE  ada0p2 (ACTIVE)
                                ada1p2 (ACTIVE)
                                ada2p2 (ACTIVE)
  mirror/wiggum-swap  COMPLETE  ada0p3 (ACTIVE)
                                ada1p3 (ACTIVE)
                                ada2p3 (ACTIVE)

That’s the main setup; Software configuration (such as nfs exports and iSCSI target) is for another blog entry. :)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s