Migrating ZFS datasets

Once your new cluster is up and running and you want to move over data sets from an old server/node to a new one, how do you do that? The method I’ve come up with is the following…..

Snapshot all datasets

First we’re going to make an initial snapshot, replicate that and then once you are ready to move over to the new server we’re going to do sync a final snapshot and then you’re over!

Create snapshot and receiving dataset

To create a snapshot of a dataset it’s pretty easy, the following command makes a snapshot named “firstsend” of dataset “videoarchive”.

zfs snapshot hddmirror/videoarchive@firstsend

On the new system we create a ZFS dataset that’s going to receive it:

zfs create hddpool/videoarchive

*note we’re sending from pool hddmirror on the old server to pool hddpool on the new server. You can also change the dataset names if you wish!

Sending the first snapshot

Most often ZFS snapshots are sent over SSH but this adds a lot of extra encryption and thus CPU workload and using 10Gbit networking between the nodes, this could impose limits. I don’t really care about encryption over a local network link so let’s use a different method using “mbuffer”!

On the new server, the receiving end we enter:

mbuffer -s 1M -m 4G -I 9090 | zfs receive -sF hddpool/videoarchive

This defines mbuffer to use 1MB buffers in size and take 4GB of memory to store these buffers if the storage isn’t ready to write it immediately yet.

Then on the old, sending server:

zfs send -v hddmirror/videoarchive@firstsend | mbuffer -s 1M -m 4G -O 10.10.10.11:9090

This again uses 1MB buffers and takes 4GB of memory to store these if the receiving system isn’t ready to receive them yet, the process in the keeps reading data, until all buffers are full.

Using all of these buffers and no encryption it allows to drive the originating storage, the network and the destination storage to perform at maximum performance at all times. Even when it hits some small files for a little bit and then some large files again, the buffers help even this out so things don’t really slow down that much!

Here are some real-world stats I could see:

in @ 1041 MiB/s, out @  681 MiB/s,  796 GiB total, buffer  14% full
in @  549 MiB/s, out @ 1041 MiB/s,  821 GiB total, buffer   5% full
in @ 1123 MiB/s, out @ 1123 MiB/s,  840 GiB total, buffer   0% full

This worked really well and helped me move my 60TB+ amount of data quite swiftly over the 10Gbit link! The buffer would stay on 0% most of the time but sometimes actually filled up to 100% on one of the sides but then either the origin storage or the destination storage would speed up again and empty the buffers. I don’t have a quantified number how much they helped, but I often saw them going up and down, keeping performance up so they certainly didn’t hurt! Another advantage is that with mbuffer, you don’t need to have SSH keys exchanged, etc.

Sending the final snapshot

Once all “firstsend” snapshots are done and you are ready to move over we send a last snapshot and then delete the snapshots on the destination storage!

One the sending side we make a new snapshot called “finalsend”:

zfs snapshot hddmirror/videoarchive@finalsend

you can browse snapshots with:

zfs list -rt all

to send it we repeat the previous steps again but with slightly different options, on the receiving side:

mbuffer -s 1M -m 16G -I 9090 | zfs receive -sF hddpool/videoarchive

and on the sending side:

zfs send -v -I hddmirror/videoarchive@firstsend hddmirror/videoarchive@finalsend | mbuffer -s 1M -m 8G -O 10.10.10.12:9090

this will automatically integrate the snapshots on the receiving end and become the “active” dataset. So on the destination end we can then do:

zfs destroy hddpool/videoarchive@firstsend
zfs destroy hddpool/videoarchive@finalsend

to delete these snapshots and we’re all done, all data is active and available on the destination server!

Conclusion

You then repeat the above commands for each of your datasets and then you are all moved over! You can move to a pool with a different name and even move to a different dataset name, all no problem!

Oh, if you ever want to rename a dataset, that’s easy:

zfs rename hddpool/videoarchive hddpool/videoarchive_old

There might be easier ways to do all of the above, especially if you have a lot of datasets, it’s quite a bit of work but I personally liked doing it this manual way which was easy to keep under control!