Site Archives zfs

FreeBSD: Go from gmirror/gjournal to ZFS

  • Posted by Chronos
  • December 24, 2007

So here’s the situation:

On one of my servers, I have a gmirror array of two 500GB hard drives with a gjournal slice. I have run out of space, so I bought two 750GB hard drives to replace these ones (I only have 4 HD bays in my 1U rackmount, so adding to it is not possible). Normally, the simple solution would be:

  • detach one drive from the array and attach one of the new ones (we’ll call it B)
  • make a new gmirror array with a new bigger-sized gjournaled-ufs slice on that B drive
  • transfer the stuff over (from A to B)
  • replace the A drive with a new one
  • rebuild the array

But it is actually still complicated. Had I previously configured my array with ZFS, the steps would be as following:

  • replace B drive with bigger capacity B drive
  • ‘grow’ zfs and let rebuild
  • replace A drive with bigger capacity A drive
  • ‘resilver’ the new drive

With ZFS, there are a lot less commands to enter too.

Now I’m going to transfer my array to ZFS so that in the future, my upgrades will be easier. I might as well do a performance check.

Step 1

# gmirror status

Name         Status     Components
mirror/gm0   COMPLETE   ad4
                        ad6
mirror/gm1   COMPLETE   ad8
                        ad10

gm1 is the array I’m upgrading. I’ll remind myself what’s on that array so I can make a correct transition later:

# grep gm1 /etc/fstab

/dev/mirror/gm1s1b          none            swap   sw        0    0
/dev/mirror/gm1s1e          /mnt/volume3    ufs    rw        2    2
/dev/mirror/gm1s1d.journal  /mnt/volume2    ufs    rw,async  2    2

# gmirror remove -v gm1 ad10

Step 2

I removed ad10 (this also tested FreeBSD’s ability to hotswap) and replaced the 500GB with the 750GB. It worked :) Then I created the ZFS pool and two mountpoints:

# zpool create tank ad10
# zfs create tank/public
# zfs create tank/private

Step 3

I then transfered all my stuff over from /mnt/volume2 and /mnt/volume3 to my newly created tank/public and tank/private storage areas. I also have to make sure NFS still worked right:

# zfs set sharenfs="-maproot=root -network 192.168.1.0" tank/public
# zfs set sharenfs="-maproot=root -network 192.168.1.0" tank/private

With that done, my NFS shares worked correctly with root account mapped. I then removed the old NFS entries from my /etc/exports file as they are now loaded automatically by the ZFS system.  Now for the swap space, I didn’t set one up with ZFS as there are stability issues right now, so I simply transfered my swap space to my other array.

Step 4

The remaining stuff to do is tweaking, and this is a per-machine configuration. What I found best for my server is the following added in my /boot/loader.conf:

# echo vm.kmem_size_max="805306368" >> /boot/loader.conf
# echo vm.kmem_size="805306368" >> /boot/loader.conf

Useful links

Performance table

Here’s my performance comparison table. It’s not scientific or anything, but it is an average of 3 tries per tests done while the system was idle. We must also remember that I went from a 500GB array with 16MB of cache to a higher density 750GB array with 32MB of cache. The numbers speak for themselves.

[TABLE=1]