Syncing ZFS datasets between nodes
In my cluster I use pve-zsync in 2 ways:
- Syncing an ISO store to all nodes
- Syncing some really important datasets to multiple nodes
Let’s find out why and how to do this!
The why
My cluster is built on the “modern” notion that a node should be able to die and most stuff should still keep running.
While doing this for VMs and Containers this is pretty easy to do from the GUI, you cannot do this for ZFS datasets sadly. But Proxmox does actually have a solution for this, pve-zsync!
This allows you to easily replicate ZFS datasets to one of the other nodes to keep an extra copy with a snapshot history and everything!
While this isn’t as good as a backup (I also make automated offsite backups of important data) it can still help you add extra protection for your important stuff by having multiple copies of it!
So more focus on multiple copy on different node redundancy vs local single server redundancy!
but but….
Yes I know you can run gluster or even CEPH to have true multi-node storage abilities and such but it also adds a lot of complexity and resource usage where I don’t really need it? Well that’s my opinion at least.
I guess it’s more of a hybrid approach between having something truly highly available (which I don’t really need, downtime is ok, I can fix it myself) and having multiple copies of it to keep it safe (which I do want).
ISO store
What if you have multiple proxmox nodes that you want to run installs on but you don’t want to bother copying the same ISO files and LXC templates to all of them individually. Well, let’s sync it automatically!
First we create the same dataset on all of the 3 nodes:
zfs create hddpool/iso
then once you’ve installed pve-zsync we can run a single command to make node1 sending it to node2 and node3 automatically on a schedule.
pve-zsync create sync --source 10.10.10.11:hddpool/iso --dest 10.10.10.12:hddpool --method ssh --source-user root --dest-user root --maxsnap 2 --name prox-iso-sync-node2
This will create the job that will automatically sync hddpool/iso from node1 to hddpool/iso on node2!
In my case I created another similar job to node 3. In proxmox I then set the storage (datacenter -> storage) to “shared” for iso and it then basically works automatically for all my ISO files, LXC templates, etc.. I just send them to node1 and within 15 minutes it’s available on the other 2 nodes too without having to think about it.
To look at this job or adjust the schedule it’s saved in a crontab file at “/etc/cron.d/pve-zsync”.
More copies for your important data!
The above is basically also the idea if you have a dataset with important data and want to have a copy on second server! By altering the “maxsnap” value you can tell pve-zsync to keep 10 or even 20 snapshots for instance so that if something happens you can always go back either locally or on the remote server (the snapshots are kept on both).
Conclusion
The above is a really way to create extra copies of your important data and keep it stores across your nodes. You don’t need to have similar nodes, the exact same hardware and also not do this for all of your data, making it very flexible.