FreeBSD: Filesystem Performance – The Setup

I’ve run a series of benchmarks on my prototyping server to determine performance differences between a variety of configurations:

  • Single drive UFS2
  • Single drive ZFS
  • ZFS 3-way mirror
  • ZFS stripe across 3 drives
  • ZFS RaidZ across 3 drives
  • ZFS RaidZ across 3 drives, plus a SSD as cache.

All the details on configuration and benchmark methods are below!

System Specification

Mainboard: MSI P45 Neo (v1.0)
Disk Controller:ICH10
RAM: 8GB DDR2 1066MHz
Hard drive:  Seagate Barracuda® 2TB (ST2000DM001)
ZFS Cache partition resides on a a Corsair CSSD-V64GB2 2.2 (shared with OS + SWAP)
OS info: FreeBSD lou 9.0-RELEASE-p3 FreeBSD 9.0-RELEASE-p3 #0: Tue Jun 12 02:52:29 UTC 2012     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64

Disk Info

diskinfo -t /dev/ada0
/dev/ada0
        512             # sectorsize
        2000398934016   # mediasize in bytes (1.8T)
        3907029168      # mediasize in sectors
        4096            # stripesize
        0               # stripeoffset
        3876021         # Cylinders according to firmware.
        16              # Heads according to firmware.
        63              # Sectors according to firmware.
        W##0###A        # Disk ident.

Seek times:
        Full stroke:      250 iter in   5.911129 sec =   23.645 msec
        Half stroke:      250 iter in   4.314233 sec =   17.257 msec
        Quarter stroke:   500 iter in   7.031946 sec =   14.064 msec
        Short forward:    400 iter in   1.652751 sec =    4.132 msec
        Short backward:   400 iter in   2.368759 sec =    5.922 msec
        Seq outer:       2048 iter in   0.322749 sec =    0.158 msec
        Seq inner:       2048 iter in   0.237719 sec =    0.116 msec
Transfer rates:
        outside:       102400 kbytes in   0.638196 sec =   160452 kbytes/sec
        middle:        102400 kbytes in   0.579845 sec =   176599 kbytes/sec
        inside:        102400 kbytes in   1.054388 sec =    97118 kbytes/sec

Tests

Machine is rebooted before each individual test. Test files are removed after each batch.

First batch: 8GiB working set, 4KiB buffer

dd RandomWrite: dd if=/dev/random bs=4k count=2m of=./test.bin
dd Read: dd if=./test.bin bs=4k count=2m of=/dev/null
dd ReadWrite: dd if=./test.bin bs=4k count=2m of=./test2.bin
dd ZeroWrite: dd if=/dev/zero bs=4k count=2m of=./test3.bin

Second batch: 8GiB working set, 64KiB buffer

dd RandomWrite: dd if=/dev/random bs=64k count=128k of=./test.bin
dd Read: dd if=./test.bin bs=64k count=128k of=/dev/null
dd ReadWrite: dd if=./test.bin bs=64k count=128k of=./test2.bin
dd ZeroWrite: dd if=/dev/zero bs=64k count=128k of=./test3.bin

Third batch: 16GiB working set

Bonnie: bonnie++ -s 16384 -n 100 -f 20M -q -m “<benchmark_name>” >>~/bonnie.csv

Configurations

UFS

gpart add -t freebsd-ufs -a 1m -l test ada0
ada0p1 added
newfs /dev/gpt/test
Reduced frags per cylinder group from 189440 to 189432 to enlarge last cyl group
/dev/gpt/test: 1907728.0MB (3907026944 sectors) block size 32768, fragment size 4096
        using 2579 cylinder groups of 739.97MB, 23679 blks, 47360 inodes.
super-block backups (for fsck -b #) at:
[many numbers]
gpart show ada0
=>        34  3907029101  ada0  GPT  (1.8T)
          34        2014        - free -  (1M)
        2048  3907026944     1  freebsd-ufs  (1.8T)
  3907028992         143        - free -  (71k)
gpart list ada0
Geom name: ada0
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 3907029134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada0p1
   Mediasize: 2000397795328 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   rawuuid: ####
   rawtype: ####
   label: test
   length: 2000397795328
   offset: 1048576
   type: freebsd-ufs
   index: 1
   end: 3907028991
   start: 2048
Consumers:
1. Name: ada0
   Mediasize: 2000398934016 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0

ZFS

gpart add -t freebsd-zfs -a 1m -l test ada0
gpart show ada0
=> 34 3907029101 ada0 GPT (1.8T)
 34 2014 - free - (1M)
 2048 3907026944 1 freebsd-zfs (1.8T)
 3907028992 143 - free - (71k)
gnop create -S 4K /dev/gpt/test
zpool create test /dev/gpt/test.nop
zfs set atime=off test
zfs set checksum=fletcher4 test
zdb | grep ashift
                ashift: 12
zpool export test
gnop destroy /dev/gpt/test.nop
zpool import -d /dev/gpt test
zpool status
  pool: test
 state: ONLINE
 scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        test        ONLINE       0     0     0
          gpt/test  ONLINE       0     0     0

errors: No known data errors

ZFS Mirror

gpart add -t freebsd-zfs -a 1m -l test1 ada0
gpart add -t freebsd-zfs -a 1m -l test2 ada1
gpart add -t freebsd-zfs -a 1m -l test3 ada2
gpart show ada0
=> 34 3907029101 ada0 GPT (1.8T)
 34 2014 - free - (1M)
 2048 3907026944 1 freebsd-zfs (1.8T)
 3907028992 143 - free - (71k)
gpart show ada1
=>        34  3907029101  ada1  GPT  (1.8T)
          34        2014        - free -  (1M)
        2048  3907026944     1  freebsd-zfs  (1.8T)
  3907028992         143        - free -  (71k)

gpart show ada2
=>        34  3907029101  ada2  GPT  (1.8T)
          34        2014        - free -  (1M)
        2048  3907026944     1  freebsd-zfs  (1.8T)
  3907028992         143        - free -  (71k)

gnop create -S 4K /dev/gpt/test1
gnop create -S 4K /dev/gpt/test2
gnop create -S 4K /dev/gpt/test3
zpool create test mirror /dev/gpt/test1.nop /dev/gpt/test2.nop /dev/gpt/test3.nop
zfs set atime=off test
zfs set checksum=fletcher4 test
zdb | grep ashift
                ashift: 12
zpool export test
gnop destroy /dev/gpt/test1.nop
gnop destroy /dev/gpt/test2.nop
gnop destroy /dev/gpt/test3.nop
zpool import -d /dev/gpt test
zpool status
  pool: test
 state: ONLINE
 scan: none requested
config:

        NAME           STATE     READ WRITE CKSUM
        test           ONLINE       0     0     0
          mirror-0     ONLINE       0     0     0
            gpt/test1  ONLINE       0     0     0
            gpt/test2  ONLINE       0     0     0
            gpt/test3  ONLINE       0     0     0

errors: No known data errors

ZFS Stripe

gpart add -t freebsd-zfs -a 1m -l test1 ada0
gpart add -t freebsd-zfs -a 1m -l test2 ada1
gpart add -t freebsd-zfs -a 1m -l test3 ada2
gpart show ada0
=> 34 3907029101 ada0 GPT (1.8T)
 34 2014 - free - (1M)
 2048 3907026944 1 freebsd-zfs (1.8T)
 3907028992 143 - free - (71k)
gpart show ada1
=>        34  3907029101  ada1  GPT  (1.8T)
          34        2014        - free -  (1M)
        2048  3907026944     1  freebsd-zfs  (1.8T)
  3907028992         143        - free -  (71k)

gpart show ada2
=>        34  3907029101  ada2  GPT  (1.8T)
          34        2014        - free -  (1M)
        2048  3907026944     1  freebsd-zfs  (1.8T)
  3907028992         143        - free -  (71k)

gnop create -S 4K /dev/gpt/test1
gnop create -S 4K /dev/gpt/test2
gnop create -S 4K /dev/gpt/test3
zpool create test /dev/gpt/test1.nop /dev/gpt/test2.nop /dev/gpt/test3.nop
zfs set atime=off test
zfs set checksum=fletcher4 test
zdb | grep ashift
                ashift: 12
                ashift: 12
                ashift: 12
zpool export test
gnop destroy /dev/gpt/test1.nop
gnop destroy /dev/gpt/test2.nop
gnop destroy /dev/gpt/test3.nop
zpool import -d /dev/gpt test
zpool status
  pool: test
 state: ONLINE
 scan: none requested
config:

        NAME         STATE     READ WRITE CKSUM
        test         ONLINE       0     0     0
          gpt/test1  ONLINE       0     0     0
          gpt/test2  ONLINE       0     0     0
          gpt/test3  ONLINE       0     0     0

errors: No known data errors

ZFS RaidZ

gpart add -t freebsd-zfs -a 1m -l test1 ada0
gpart add -t freebsd-zfs -a 1m -l test2 ada1
gpart add -t freebsd-zfs -a 1m -l test3 ada2
gpart show ada0
=> 34 3907029101 ada0 GPT (1.8T)
 34 2014 - free - (1M)
 2048 3907026944 1 freebsd-zfs (1.8T)
 3907028992 143 - free - (71k)
gpart show ada1
=>        34  3907029101  ada1  GPT  (1.8T)
          34        2014        - free -  (1M)
        2048  3907026944     1  freebsd-zfs  (1.8T)
  3907028992         143        - free -  (71k)

gpart show ada2
=>        34  3907029101  ada2  GPT  (1.8T)
          34        2014        - free -  (1M)
        2048  3907026944     1  freebsd-zfs  (1.8T)
  3907028992         143        - free -  (71k)

gnop create -S 4K /dev/gpt/test1
gnop create -S 4K /dev/gpt/test2
gnop create -S 4K /dev/gpt/test3
zpool create test raidz /dev/gpt/test1.nop /dev/gpt/test2.nop /dev/gpt/test3.nop
zfs set atime=off test
zfs set checksum=fletcher4 test
zdb | grep ashift
                ashift: 12
zpool export test
gnop destroy /dev/gpt/test1.nop
gnop destroy /dev/gpt/test2.nop
gnop destroy /dev/gpt/test3.nop
zpool import -d /dev/gpt test
zpool status
  pool: test
 state: ONLINE
 scan: none requested
config:

        NAME           STATE     READ WRITE CKSUM
        test           ONLINE       0     0     0
          raidz1-0     ONLINE       0     0     0
            gpt/test1  ONLINE       0     0     0
            gpt/test2  ONLINE       0     0     0
            gpt/test3  ONLINE       0     0     0

errors: No known data errors

ZFS RaidZ + Cache

eSame commands as ZFS RaidZ, plus these:

gpart add -a 1m -t freebsd-zfs -l zfscache ada3
gpart show ada3
=>       34  125045357  ada3  GPT  (59G)
         34        128     1  freebsd-boot  (64k)
        162   16777216     2  freebsd-ufs  (8.0G)
   16777378    4194304     3  freebsd-swap  (2.0G)
   20971682       1886        - free -  (943k)
   20973568  104071168     4  freebsd-zfs  (49G)
  125044736        655        - free -  (327k)
zpool add test cache /dev/gpt/zfscache
zpool status
  pool: test
 state: ONLINE
 scan: none requested
config:

        NAME            STATE     READ WRITE CKSUM
        test            ONLINE       0     0     0
          raidz1-0      ONLINE       0     0     0
            gpt/test1   ONLINE       0     0     0
            gpt/test2   ONLINE       0     0     0
            gpt/test3   ONLINE       0     0     0
        cache
          gpt/zfscache  ONLINE       0     0     0

errors: No known data errors

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s