I’ve run a series of benchmarks on my prototyping server to determine performance differences between a variety of configurations:
- Single drive UFS2
- Single drive ZFS
- ZFS 3-way mirror
- ZFS stripe across 3 drives
- ZFS RaidZ across 3 drives
- ZFS RaidZ across 3 drives, plus a SSD as cache.
All the details on configuration and benchmark methods are below!
System Specification
Mainboard: MSI P45 Neo (v1.0)
Disk Controller:ICH10
RAM: 8GB DDR2 1066MHz
Hard drive: Seagate Barracuda® 2TB (ST2000DM001)
ZFS Cache partition resides on a a Corsair CSSD-V64GB2 2.2 (shared with OS + SWAP)
OS info: FreeBSD lou 9.0-RELEASE-p3 FreeBSD 9.0-RELEASE-p3 #0: Tue Jun 12 02:52:29 UTC 2012 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
Disk Info
diskinfo -t /dev/ada0
/dev/ada0
512 # sectorsize
2000398934016 # mediasize in bytes (1.8T)
3907029168 # mediasize in sectors
4096 # stripesize
0 # stripeoffset
3876021 # Cylinders according to firmware.
16 # Heads according to firmware.
63 # Sectors according to firmware.
W##0###A # Disk ident.
Seek times:
Full stroke: 250 iter in 5.911129 sec = 23.645 msec
Half stroke: 250 iter in 4.314233 sec = 17.257 msec
Quarter stroke: 500 iter in 7.031946 sec = 14.064 msec
Short forward: 400 iter in 1.652751 sec = 4.132 msec
Short backward: 400 iter in 2.368759 sec = 5.922 msec
Seq outer: 2048 iter in 0.322749 sec = 0.158 msec
Seq inner: 2048 iter in 0.237719 sec = 0.116 msec
Transfer rates:
outside: 102400 kbytes in 0.638196 sec = 160452 kbytes/sec
middle: 102400 kbytes in 0.579845 sec = 176599 kbytes/sec
inside: 102400 kbytes in 1.054388 sec = 97118 kbytes/sec
Tests
Machine is rebooted before each individual test. Test files are removed after each batch.
First batch: 8GiB working set, 4KiB buffer
dd RandomWrite: dd if=/dev/random bs=4k count=2m of=./test.bin
dd Read: dd if=./test.bin bs=4k count=2m of=/dev/null
dd ReadWrite: dd if=./test.bin bs=4k count=2m of=./test2.bin
dd ZeroWrite: dd if=/dev/zero bs=4k count=2m of=./test3.bin
Second batch: 8GiB working set, 64KiB buffer
dd RandomWrite: dd if=/dev/random bs=64k count=128k of=./test.bin
dd Read: dd if=./test.bin bs=64k count=128k of=/dev/null
dd ReadWrite: dd if=./test.bin bs=64k count=128k of=./test2.bin
dd ZeroWrite: dd if=/dev/zero bs=64k count=128k of=./test3.bin
Third batch: 16GiB working set
Bonnie: bonnie++ -s 16384 -n 100 -f 20M -q -m “<benchmark_name>” >>~/bonnie.csv
Configurations
UFS
gpart add -t freebsd-ufs -a 1m -l test ada0
ada0p1 added
newfs /dev/gpt/test
Reduced frags per cylinder group from 189440 to 189432 to enlarge last cyl group
/dev/gpt/test: 1907728.0MB (3907026944 sectors) block size 32768, fragment size 4096
using 2579 cylinder groups of 739.97MB, 23679 blks, 47360 inodes.
super-block backups (for fsck -b #) at:
[many numbers]
gpart show ada0
=> 34 3907029101 ada0 GPT (1.8T)
34 2014 - free - (1M)
2048 3907026944 1 freebsd-ufs (1.8T)
3907028992 143 - free - (71k)
gpart list ada0
Geom name: ada0
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 3907029134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada0p1
Mediasize: 2000397795328 (1.8T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r0w0e0
rawuuid: ####
rawtype: ####
label: test
length: 2000397795328
offset: 1048576
type: freebsd-ufs
index: 1
end: 3907028991
start: 2048
Consumers:
1. Name: ada0
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r0w0e0
ZFS
gpart add -t freebsd-zfs -a 1m -l test ada0
gpart show ada0
=> 34 3907029101 ada0 GPT (1.8T)
34 2014 - free - (1M)
2048 3907026944 1 freebsd-zfs (1.8T)
3907028992 143 - free - (71k)
gnop create -S 4K /dev/gpt/test
zpool create test /dev/gpt/test.nop
zfs set atime=off test
zfs set checksum=fletcher4 test
zdb | grep ashift
ashift: 12
zpool export test
gnop destroy /dev/gpt/test.nop
zpool import -d /dev/gpt test
zpool status
pool: test
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
test ONLINE 0 0 0
gpt/test ONLINE 0 0 0
errors: No known data errors
ZFS Mirror
gpart add -t freebsd-zfs -a 1m -l test1 ada0
gpart add -t freebsd-zfs -a 1m -l test2 ada1
gpart add -t freebsd-zfs -a 1m -l test3 ada2
gpart show ada0
=> 34 3907029101 ada0 GPT (1.8T)
34 2014 - free - (1M)
2048 3907026944 1 freebsd-zfs (1.8T)
3907028992 143 - free - (71k)
gpart show ada1
=> 34 3907029101 ada1 GPT (1.8T)
34 2014 - free - (1M)
2048 3907026944 1 freebsd-zfs (1.8T)
3907028992 143 - free - (71k)
gpart show ada2
=> 34 3907029101 ada2 GPT (1.8T)
34 2014 - free - (1M)
2048 3907026944 1 freebsd-zfs (1.8T)
3907028992 143 - free - (71k)
gnop create -S 4K /dev/gpt/test1
gnop create -S 4K /dev/gpt/test2
gnop create -S 4K /dev/gpt/test3
zpool create test mirror /dev/gpt/test1.nop /dev/gpt/test2.nop /dev/gpt/test3.nop
zfs set atime=off test
zfs set checksum=fletcher4 test
zdb | grep ashift
ashift: 12
zpool export test
gnop destroy /dev/gpt/test1.nop
gnop destroy /dev/gpt/test2.nop
gnop destroy /dev/gpt/test3.nop
zpool import -d /dev/gpt test
zpool status
pool: test
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
test ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gpt/test1 ONLINE 0 0 0
gpt/test2 ONLINE 0 0 0
gpt/test3 ONLINE 0 0 0
errors: No known data errors
ZFS Stripe
gpart add -t freebsd-zfs -a 1m -l test1 ada0
gpart add -t freebsd-zfs -a 1m -l test2 ada1
gpart add -t freebsd-zfs -a 1m -l test3 ada2
gpart show ada0
=> 34 3907029101 ada0 GPT (1.8T)
34 2014 - free - (1M)
2048 3907026944 1 freebsd-zfs (1.8T)
3907028992 143 - free - (71k)
gpart show ada1
=> 34 3907029101 ada1 GPT (1.8T)
34 2014 - free - (1M)
2048 3907026944 1 freebsd-zfs (1.8T)
3907028992 143 - free - (71k)
gpart show ada2
=> 34 3907029101 ada2 GPT (1.8T)
34 2014 - free - (1M)
2048 3907026944 1 freebsd-zfs (1.8T)
3907028992 143 - free - (71k)
gnop create -S 4K /dev/gpt/test1
gnop create -S 4K /dev/gpt/test2
gnop create -S 4K /dev/gpt/test3
zpool create test /dev/gpt/test1.nop /dev/gpt/test2.nop /dev/gpt/test3.nop
zfs set atime=off test
zfs set checksum=fletcher4 test
zdb | grep ashift
ashift: 12
ashift: 12
ashift: 12
zpool export test
gnop destroy /dev/gpt/test1.nop
gnop destroy /dev/gpt/test2.nop
gnop destroy /dev/gpt/test3.nop
zpool import -d /dev/gpt test
zpool status
pool: test
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
test ONLINE 0 0 0
gpt/test1 ONLINE 0 0 0
gpt/test2 ONLINE 0 0 0
gpt/test3 ONLINE 0 0 0
errors: No known data errors
ZFS RaidZ
gpart add -t freebsd-zfs -a 1m -l test1 ada0
gpart add -t freebsd-zfs -a 1m -l test2 ada1
gpart add -t freebsd-zfs -a 1m -l test3 ada2
gpart show ada0
=> 34 3907029101 ada0 GPT (1.8T)
34 2014 - free - (1M)
2048 3907026944 1 freebsd-zfs (1.8T)
3907028992 143 - free - (71k)
gpart show ada1
=> 34 3907029101 ada1 GPT (1.8T)
34 2014 - free - (1M)
2048 3907026944 1 freebsd-zfs (1.8T)
3907028992 143 - free - (71k)
gpart show ada2
=> 34 3907029101 ada2 GPT (1.8T)
34 2014 - free - (1M)
2048 3907026944 1 freebsd-zfs (1.8T)
3907028992 143 - free - (71k)
gnop create -S 4K /dev/gpt/test1
gnop create -S 4K /dev/gpt/test2
gnop create -S 4K /dev/gpt/test3
zpool create test raidz /dev/gpt/test1.nop /dev/gpt/test2.nop /dev/gpt/test3.nop
zfs set atime=off test
zfs set checksum=fletcher4 test
zdb | grep ashift
ashift: 12
zpool export test
gnop destroy /dev/gpt/test1.nop
gnop destroy /dev/gpt/test2.nop
gnop destroy /dev/gpt/test3.nop
zpool import -d /dev/gpt test
zpool status
pool: test
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
test ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
gpt/test1 ONLINE 0 0 0
gpt/test2 ONLINE 0 0 0
gpt/test3 ONLINE 0 0 0
errors: No known data errors
ZFS RaidZ + Cache
eSame commands as ZFS RaidZ, plus these:
gpart add -a 1m -t freebsd-zfs -l zfscache ada3
gpart show ada3
=> 34 125045357 ada3 GPT (59G)
34 128 1 freebsd-boot (64k)
162 16777216 2 freebsd-ufs (8.0G)
16777378 4194304 3 freebsd-swap (2.0G)
20971682 1886 - free - (943k)
20973568 104071168 4 freebsd-zfs (49G)
125044736 655 - free - (327k)
zpool add test cache /dev/gpt/zfscache
zpool status
pool: test
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
test ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
gpt/test1 ONLINE 0 0 0
gpt/test2 ONLINE 0 0 0
gpt/test3 ONLINE 0 0 0
cache
gpt/zfscache ONLINE 0 0 0
errors: No known data errors
Savagedlight's Road 