Partly a cluster, partly not?!
Home Server Cluster 2025 index
Over the years I have been using Proxmox and I’ve grown to like it a lot. I know there are a lot of other solutions out there but Proxmox takes the right boxes for me such as being heavily integrated with ZFS and allowing you to run Clusters even with dissimilar hardware, etc. so I will keep using it moving forward.
Cluster with no shared storage?
One of the things I have transitioned to a few years back on my current Proxmox cluster is not running my VMs on storage that is RAID backed up or at least not on all nodes. Instead I use a different strategy which running in a cluster enables.
ZFS replication!
A while back Proxmox introduced ZFS replication where you can have the same ZFS pool configured on each node individually. This can be a single SSD or a mirror. You can then configure a virtual machine to be replicated to another node or even multiple nodes! This snapshot replication automatically happens every 15 minutes for instance.
Multiple copies inside of the cluster!
This means having a Mirror or RAID storage per node has suddenly become a lot less important for this kind of expensive storage! Instead of keeping multiple copies of the VM on local storage you can use ZFS replication to send the changes to the other 2 nodes in the cluster every 15 minutes.
This way, although a single node only has 1 copy you still have 2 other copies on the other 2 nodes! So even if the node the VM is running on fails and it’s started again from one of the other nodes, that version of the virtual machine will only be behind 0 to 15 minutes of what was running.
For my purposes, such as keeping Home Assistant available, those 15 minutes really don’t matter so much to me that it’s worth running shared storage or some other kind of solution. I can loose a bit of sensor data, it’s not that important to me.
What about other Cluster functionality such as live migrations?
The brilliant thing is that things like moving a Virtual Machine between nodes still remains possible. If you ask a VM to be moved from Node A to Node B Proxmox takes a quick snapshot, sends that over to the other node and then starts a sort of live-sync over the network while it also transfers the content of the memory and starts keeping that in sync, the at one point in time the VM is switched over all while running and with minimal interruption!
So even though there isn’t any shared storage that kind of functionality is still available to you. It is recommend to have some kind of fast storage and it depends a little bit on how busy your VMs are but from my experience, running like this for over 2 years now, it works great and with faster hardware (like 10Gbit on all nodes), it’ll only become better!
Experimentation!
I might experiment with cluster storage in the future but more as a case of “I wonder how this works”, something like LinBit HA which is an Open Source storage layer you can hook into Proxmox does sound interesting to me! Functionality wise I don’t really need it for my needs above what is in Proxmox itself and if it for instance would be a bother to maintain, it would be just for experimentation only.
So no Storage clustering?
No I think I actually won’t have any clustered highly available storage. Although I want certain compute to be highly available (such as my Home Assistant installs), I don’t really have the same need for my bulk storage. I’m fine with a volume being down if a server would fail. In worst case I have some old hardware lying around I could swap the disks over to and bring them online (very easy with ZFS) to access what I need.
I do still want to experiment with it though, something like having certain ZFS datasets that get replicated for instance could be interesting for very important or “hot” data I’m working on at that moment.
Or even full blown clustered storage on top of ZFS, kind of like LinBit also does. We’ll see in the future, currently it’s not part of the design of the setup though.