I’ve recently upgraded Wiggum (my file server) from FreeBSD 8.0 to FreeBSD 9.0. Since I had made some mistakes when originally setting up Wiggum two years ago, I went for a complete reinstall – and recreation of the zpools. This blog entry is a step-by-step guide for how I did the initial installation and setup.
Specifications
- CPU: Intel Core i7 930 @ 2.8GHx (no change)
- Mainboard: Gigabyte GA-X58A-UD3R (no change)
- PSU: Cooler Master Silent Pro M600 600W (no change)
- RAM: Corsair XMS3 DDR3 1600MHz 12GB CL9 (no change)
- Extra SATA: HighPoint RocketRAID 2320 8P SATA (no change)
- NIC: 2x Intel Pro/1000CT PCIe Desktop Adapter (no change)
- Storage:
- 3x Intel® SSD 330 Series 120GB, which will be used for OS, swap, ZFS log and ZFS cache
- 12x 1.5TB HDD (6x WDC GreenPower and 6x Seagate Barracuda 7200.11), which will be used for zpools.
Partitioning
System
The SSDs are partitioned with a boot sector, 8GB for OS, 1GB for swap, 2x 6GB for ZFS log, 10+80GB for ZFS cache. OS and swap are 3-way GEOM mirrors, while ZFS log is a 3-way ZFS mirror. Cache has no mirroring.
I booted the FreeBSD 9.0 AMD64 install CD (disk1), went through the installer as normal until asked about partitioning. Here, I chose to enter shell, and executed the following commands:
# Support for geom mirroring kldload geom_mirror ### Start: For each system drive: gpart create -s gpt da0 gpart add -t freebsd-boot -l Bay1.X-boot -s 64 da0 gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 da0 gpart add -a 4m -t freebsd-ufs -l Bay1.X-system -s 8G da0 gpart add -a 4m -t freebsd-swap -l Bay1.X-swap -s 1G da0 # log device should be half of system ram at most, according to http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide#Separate_Log_Devices # log for backup pool gpart add -a 4m -t freebsd-zfs -l Bay1.X-log1 -s 6G da0 # log for storage pool gpart add -a 4m -t freebsd-zfs -l Bay1.X-log2 -s 6G da0 # Cache for backup pool gpart add -a 4m -t freebsd-zfs -l Bay1.X-cache1 -s 10G da0 # Cache for storage pool gpart add -a 4m -t freebsd-zfs -l Bay1.X-cache2 -s 80G da0 ### END: For each system drive # Partitions are now a-okay. Create geom mirrors! gmirror label -v -b round-robin wiggum-system /dev/gpt/Bay1.1-system /dev/gpt/Bay1.2-system /dev/gpt/Bay1.3-system gmirror label -v -b round-robin wiggum-swap /dev/gpt/Bay1.1-swap /dev/gpt/Bay1.2-swap /dev/gpt/Bay1.3-swap # Format the system partition newfs -U -t -S 4k /dev/mirror/wiggum-system # Mount the system partition where the installer expects it mount /dev/mirror/wiggum-system /mnt # Create a fstab which will be installed by the installer echo "/dev/mirror/wiggum-swap none swap sw 0 0" > /tmp/bsdinstall_etc/fstab echo "/dev/mirror/wiggum-system / ufs rw 1 1" >> /tmp/bsdinstall_etc/fstab # Resume installer by typing "exit"
After the installer is done, it will ask you if you want to make any final changes in a shell or reboot. Opt to use the shell. This shell will be chrooted into your new OS.
Execute the following command in the shell:
echo 'geom_mirror_load="YES"' >>/boot/loader.config
Data
After booting the system, I removed the partition tables on the data drives, then partitioned them as follows:
gpart create -s gpt adaX gpart add -a 4k -t freebsd-zfs -l "BayX.Y adaX
and then, I created the pools:
gnop create -S 4k /dev/gpt/Bay* zpool create backup raidz1 gpt/Bay2.2.nop gpt/Bay2.3.nop gpt/Bay3.*.nop log mirror gpt/Bay1.*-log1.nop cache gpt/Bay1.*-cache1 zpool create storage raidz2 gpt/Bay4.*.nop gpt/Bay5.*.nop log mirror gpt/Bay1.*-log2.nop cache gpt/Bay1.*-cache2 spare gpt/Bay2.1.nop zpool export backup zpool export storage gnop destroy /dev/gpt/*.nop zpool import -d /dev/gpt backup zpool import -d /dev/gpt storage
And the end result (retrieved a week later):
zpool status
pool: backup
state: ONLINE
scan: scrub repaired 0 in 0h10m with 0 errors on Wed Aug 29 01:10:02 2012
config:
NAME STATE READ WRITE CKSUM
backup ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
gpt/Bay2.2-Seagate-XXXXXXXX ONLINE 0 0 0
gpt/Bay2.3-Seagate-XXXXXXXX ONLINE 0 0 0
gpt/Bay3.1-WDCGP-XXXXXXXXXXXX ONLINE 0 0 0
gpt/Bay3.2-WDCGP-XXXXXXXXXXXX ONLINE 0 0 0
gpt/Bay3.3-WDCGP-XXXXXXXXXXXX ONLINE 0 0 0
logs
mirror-1 ONLINE 0 0 0
gpt/Bay1.1-log1 ONLINE 0 0 0
gpt/Bay1.2-log1 ONLINE 0 0 0
gpt/Bay1.3-log1 ONLINE 0 0 0
cache
gpt/Bay1.1-cache1 ONLINE 0 0 0
gpt/Bay1.2-cache1 ONLINE 0 0 0
gpt/Bay1.3-cache1 ONLINE 0 0 0
errors: No known data errors
pool: storage
state: ONLINE
scan: scrub repaired 32K in 3h52m with 0 errors on Sun Aug 26 04:29:51 2012
config:
NAME STATE READ WRITE CKSUM
storage ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gpt/Bay4.1-Seagate-XXXXXXXX ONLINE 0 0 0
gpt/Bay4.2-Seagate-XXXXXXXX ONLINE 0 0 0
gpt/Bay4.3-Seagate-XXXXXXXX ONLINE 0 0 0
gpt/Bay5.1-WDCGP-XXXXXXXXXXXX ONLINE 0 0 0
gpt/Bay5.2-WDCGP-XXXXXXXXXXXX ONLINE 0 0 0
gpt/Bay5.3-WDCGP-XXXXXXXXXXXX ONLINE 0 0 0
logs
mirror-1 ONLINE 0 0 0
gpt/Bay1.1-log2 ONLINE 0 0 0
gpt/Bay1.2-log2 ONLINE 0 0 0
gpt/Bay1.3-log2 ONLINE 0 0 0
cache
gpt/Bay1.1-cache2 ONLINE 0 0 0
gpt/Bay1.2-cache2 ONLINE 0 0 0
gpt/Bay1.3-cache2 ONLINE 0 0 0
spares
gpt/Bay2.1-Seagate-9VS44MX8 AVAIL
errors: No known data errors
gmirror status
Name Status Components
mirror/wiggum-system COMPLETE ada0p2 (ACTIVE)
ada1p2 (ACTIVE)
ada2p2 (ACTIVE)
mirror/wiggum-swap COMPLETE ada0p3 (ACTIVE)
ada1p3 (ACTIVE)
ada2p3 (ACTIVE)
That’s the main setup; Software configuration (such as nfs exports and iSCSI target) is for another blog entry. :)
Savagedlight's Road 