How to install Proxmox and setup a ZFS Pool
This article is to accompany my video about setting up Proxmox, creating a ZFS Pool and then installing a small VM on it. It’s a tutorial from start to finish!
Part of multiple articles
The article index for this 100TB 10Gbit server series can be found by clicking here.
How to install Proxmox and setup a ZFS Pool + VM
During this video we’re installing proxmox, creating a ZFS pool and installing a VM, here are the parts in order with the video. It should contain everything you need to accomplish the same as in the video!
Again, main explanations will be in the video, below text is to go along with it!
- Proxmox ISO file – https://www.proxmox.com/en/downloads
- BalenaEtcher – https://www.balena.io/etcher/
Some BIOS settings need to be changed for Proxmox do be able to do it’s virtualization, mainly those are:
- SVM Mode – Enabled (Enables CPU Virtualization)
- Storage Boot Option Control – Disabled (Disables loading PCIe add-in card ROMs)
- Other PCIe Device ROM Priority – Disabled
Fixing Update Repository
By Default Proxmox is configured for the Enterprise repository, without a license you can’t use those so to reconfigure it to the community repository, do the following.
cd /etc/apt/sources.list.d/ ls rm pve-enterprise.list cd ..
After this, we are going to edit the “sources.list” file and replace it’s content. You can delete lines in nano by using “ctrl+k”.
nano sources.list deb http://ftp.debian.org/debian buster main contrib deb http://ftp.debian.org/debian buster-updates main contrib # PVE pve-no-subscription repository provided by proxmox.com, # NOT recommended for production use deb http://download.proxmox.com/debian/pve buster pve-no-subscription # security updates deb http://security.debian.org buster/updates main contrib
Save the file by hitting “ctrl+x” and choosing yes
Erasing partition table on disks
First we need to delete the partitions on the disks, we do this using a tool called “gdisk” which is installed by default
First look up the disks listed in /dev/disk/by-id:
cd /dev/disk/by-id ls
Then, select the disks you want to clear and run the following command as you see in the video:
There you can clear the disk by listing the partitions with “p” and then deleting them with “d”. Once you are done deleting the partitions, exit with “w”.
Creaintg a simple RAIDz pool
To setup a RAIDz2 pool use the following command (the amount of disks can vary to your needs)
zpool create NAME -o ashift=12 raidz2 /dev/disk/by-id/DISK1 /dev/disk/by-id/DISK2 /dev/disk/by-id/DISK3 /dev/disk/by-id/DISK4 /dev/disk/by-id/DISK5
To look at the pool status run:
To turn compression on for the pool run:
zfs set compression=lz4 POOLNAME
Creating ISO storage
Here we create a dataset using the command-line using:
zfs create POOL/ISO
For people who don’t enjoy videos and would rather just read, here is the script I used while preparing for the video. It’s raw and not edited to be in an easy to read post form! If you find this useful, please let me know and I’ll try to post it more often!
Hey guys, in this video we’re going to take a look at installing proxmox. This is going to be a bit more basic level tutorial video but I have some more in depth stuff planned for upcoming videos!
In this video we’ll run through downloading the ISO and creating a USB stick. Then we are going to look at some BIOS settings and after that run through the installation process together.
Once that’s done, we are also going to run through creating a simple RAIDz pool from 5x2TB disks. In a future video we are going to dive a bit more in depth into some more advanced ZFS Mirror pools with SSD caching.
In the last part of the video we do some basic setup stuff for a new proxmox server and to end we’re going to install a small Linux VM together!
I’ll make sure to include time jump links again, Ok, let’s go!
Right, so to get started we need to download the newest version of proxmox. Links to everything and command-line commands will be put in a blog post you’ll find in the description, please check that out if you want to perform any of these steps yourself! On the download page download the ISO file, we are going to use the 6.0 version.
While that’s downloading we also need a tool called “balenaetcher”.
Once both are downloaded and etcher is installed, insert your USB stick (Anything 1GB or more should be fine) and start etcher. Point it to the downloaded ISO file and then to the USB stick, hit write and wait. 🙂
You might get some partition popups, just click cancel on those.
Ok, stick done! Let’s go to the server and check some BIOS settings to make sure we avoid some little problems.
— start server and hit del like a moron
In the BIOS it’s probably best to load optimized defaults and then make the following changes:
First we want to enable “SVM Mode”, in my BIOS that’s located under “Advanced Frequency Settings” and then “Advanced CPU Core Settings”. There you just set “SVM Mode” to enabled. This enables certain virtualization features inside of your CPU which Proxmox wants to use.
Then in the BIOS tab you want to set “Storage Boot Option Control” to Disabled and also set “Other PCI Device ROM Priority” to disabled.
This disables the loading of option ROMs during boot. Since we won’t be booting from the network or any storage connected to the LSI controller, we don’t need to load their BIOSses while booting. If you do want to boot from the LSI controller you need to set the first option to “Legacy Only”. Setting it to UEFI can work but that means you need to do certain Proxmox stuff differently and I don’t recommend that right now!
Ok, hit F10 to save all changes and hit F12 to get into the boot override menu. There you should see your USB stick listed, now it depends on your system if you need to select the UEFI version of your USB Stick or not. In my case I need to, so I select “UEFI USB Disk 2.0 PMAP”.
A thing of note, a mouse could actually be easy to use during the graphical installer part, if you have one, make sure it’s connected!
Once booted you should be greeted with the following screen. There hit the first option to Install Proxmox.
Ofcourse we agree with the license agreement 😛
Then we get to the storage configuration for where you want to install Proxmox. Here we are going to make a few changes!
Hit Options and change EXT4 to ZFS (Raid 1). As you can see all the disks Proxmox detects are now shown and we want to select the SSDs of which we want to create a mirror and install Proxmox onto. Set all the others to “– do not use –“. Then we want to do a little tweaking in the advanced options.
Only change compression from on to LZ4. This shouldn’t be necessary but I like to be sure. Then we also want to change the hdsize value. These are 256GB SSDs but I never use the full size if I can avoid it. So in this case I’m going to enter a value of 200GB.
Hit ok and Next!
Fill in your Country details and once everything is correct hit Next again.
Set a password and enter an e-mail address. This address is important because the server will send you messages when something is wrong!
Select the correct network card you connected and make sure to fill in a correct FQDN hostname like you see here. Then also fill in all the IP details, it’s best to always use a static IP for your virtual machine host machine.
Once you hit Next you’ll see a quick summary screen and once you hit Install the installation starts!
Ok, all done, let’s reboot!
Once the installation is done, Promxox should automatically boot. If it doesn’t, check your BIOS that you have the Linux BootLoader or SSDs used selected as first boot devices. Since we are running a mirror, make sure to select both so that if one fails it will still boot from the other one!
OK, that’s done. Great! Almost done with all the basics! Let’s configure a few things more, but using the web interface and remote sessions!
First we are going to make sure you can update Proxmox to the latest version. To do that we need to enable to community repository.
To do this we log into the web interface, the address and port number for this are listed on the screen. The first time you will get a security warning, you can ignore this safely. After we proceed you’ll see the proxmox login prompt, there we log in with “root” and the password used during the installer.
Every time you log into this interface you will get a pop-up that you don’t have an official license. There are ways to remove this pop-up but I don’t really condone those methods. You can use the software without paying for it and they only give you a single pop-up when you log in. If you hate the pop-up that much, buy a license. 😉
Ok, once in the interface, the first thing we want to do is get the latest updates. But this doesn’t work out of the box, you need to reconfigure proxmox to use the community repository.
To do this, open a shell and choose xterm. This didn’t work while recording so I opened a putty session instead, basically the same thing. Just connect to the IP of the proxmox host and login with root and your password again.
There we are going to change the repositories available. All these steps will also be type out in a blog article so you can repeat them more easily.
Basically, we are removing the pve-enterprise repository file and then editing the default one to include the community version.
After that we can run a “apt update” to get all the latest information and then install it using “apt dist-upgrade”.
Once that’s done, give the machine a reboot and you’re done, updating will now also work from the GUI. To make sure the GUI reflects all changes, give it a refresh once the server is back up.
Ok, next let’s configure a simple RAIDz pool. I’ll be doing videos on more complex ZFS technical details and how to create more advanced pools but for now this is fine to start out with.
My disks have already been used so although they show up under “disks” I can’t configure a pool on them, let’s fix that. Also, although configuring a pool from the GUI now is available I’d rather still do it from the command-line. That way I have a bit more control over it and I also like to use /dev/disk/by-id links instead of /dev/sda sdb etc.. Using the /dev/disk/by-id links it doesn’t matter how you reshuffle your disks, even between controllers or systems, they will always work!
Ok, let’s first clear those disks. To do that we go back to command-line and use a tool called “gdisk” to basically delete all partitions from the disks.
These are my Samsung disks I want to use. Let’s erase them. To erase you look at which partitions exist using “p” and then d both those partitions and exist using w. And then we repeat that for all the disks!
Now if we check in the GUI we can see that all disks are available, but as said, I’d rather use the command-line anyway.
Again, commands will be listed in the description if you want to be able to read them more without the video.
I’m not going to explain the zpool create command all too much. Just know you should basically always use the -o ashift=12. The “rz2TB” is the name I’m giving the pool and I’m choosing a raidZ2 which uses 2 parity disks, so basically equivalent to RAID6 protection wise. I’m using old 2TB Samsung disks which have been in service for over 8 years now, but safe then sorry. 😉
Ok, zpool created and available! Let’s make sure compression is turned on for the whole pool. LZ2 compression is basically free on modern processors so it’s good to always enable it!
Ok, the GUI is seeing our new pool and everything looks correct! But it’s not available to use yet. You still need to add it as “storage” within Proxmox. You do that by adding it in the storage menu. I enable thin provision because that works best if you want to host VM’s on it.
Ok, all done and online! Next, lets create a place where we can put ISO files and make backups to install a Ubuntu VM.
To do so we create whats called a ZFS dataset on the command-line. A dataset is a sort of mix between a partition and a directory. If you are going to create directories with a specific purpose and want to maybe share them later on or set a quota on it, it’s good to make it a dataset.
Then we are going to add that dataset as a directory to Proxmox so we can upload our Linux ISO install file to it!
In storage we now add a directory, call it ISO and point it toward the dataset on the pool. Here we say we want it to house everything except disk images and containers and we’ll allow up to 10 backups.
Ok, go to the ISO storage and upload your favorite Linux ISO file, in my case a netinstall of Ubuntu 18.04LTS. I won’t go through all the settings right now, but we’re using the ISO image just uploaded and putting it on the boot SSDs for speed.
Ok, once created, click on the VM, 100 and start it. Then you can click the console button to view it. Ok, I’ll run through a very quick install of Ubuntu!
And well, that’s it! You’ve now setup a Proxmox machine, made it so we can run updates, created some ZFS storage and even installed your first virtual machine!
Hopefully this tutorial was helpful to you. I’m quickly going to continue using my server to migrate data from the old server to the new one! This was a bit of a “basic” video to get started, more advanced and in depth videos coming up soon! A like and subscribe is always appreciated, a share even more and I hope to see you back next time! 🙂
Again, this post if part of a larger series, click here to go to the index page.
8 thoughts on “How to install Proxmox and setup a ZFS Pool”
Great info, I am following it, for my own system.
please note but there are a few UPPER/lowercase errors in the commands to create the RAID.
I also had to gdisk, then use the command “o” and the command “w” to completely empty the disks and write the info to the disk.
As I have a 24 disk system using /dev/sda to /dev/sdn (in my case) was easier for this gdisk command. And indeed I used the by-disk also, which is what you should do when you have hot swappable disks.
Thnx for the comment, good points 🙂
Excellent guide! I sincerely appreciate the time and effort you’ve put into this (and your other projects). You’ve made the command line fun again. 🙂
I used this manual to install Proxmox 7 only for the update repo there is a change in file /etc/apt/sources.list
(press A to Add / change)
deb http://ftp.debian.org/debian bullseye main contrib
deb http://ftp.debian.org/debian bullseye-updates main contrib
# PVE pve-no-subscription repository provided by proxmox.com,
# NOT recommended for production use
deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription
# security updates
deb http://security.debian.org/debian-security bullseye-security main contrib
When this is added press ESC and then Shift + Z and release Shift press enter
Can you make a youtube video on how to replace a rpool hard drive?