Proxmox setup and uses
Proxmox setup and uses
Well i was asked to write this innards up because i have talked about it alot in the last year or so and i am happy to do it. I also needed to do some research to make sure that my data was correct and i am told that i need to start practicing with AI more so i did use it to bring a lot of this data together. I did verify as much as i could and i am dont mind being corrected if anyone finds it necessary.
First what is proxmox
Proxmox is a free and open source server management platform that allows for full virtualization as well as lighterweight contanierization using a web based interface that interconnect with other proxmox ‘nodes’ and is a bare metal hypervisor that sits on top of a full debian distrobution.
Couple of things that need to be explained in that definition. You can run fully virtualized operating systems that have their own kernels and are fully isolated from everything else. The interface allows you to dedicate the amount of ram, disk space and the amount of processor to use. As well as many other things like the architecture of the processor to be emulated, the type of disk to emulate such as ssd scsi and a few other things. You can also control what gets passed to the VM in a several different ways. You can pass PCI devices to the VM with PCI passthrough which is already setup and working at install time(with exceptions). You can pass a mapped usb device or a usb port or an unmapped device depending on the need.
You can also pass full disks to a vm if you want or you can create directories from the node and just pass the directory. Or create virtual ethernet bridges that can be attached to physical ethernet ports and then force VM’s or containers to use that for their internet. Another usefull thing you can do when resources are somewhat limited is over allocate the entirety of the resources available and then set everything to ballooning so that different operations can grab the needed resources as they become available. Or limit the amount of resources that different operations can use if it is a lower priority.
Next up is containers. In this case they are LXC’s which are lightweight containers that share the host systems kernel instead of having one of their own making them fast and efficient for running specific services. They have taken the place of a lot of my previous Docker build outs but we can get back to that. While it is a bit more difficult in some cases you can still pass everything to these that you can to a VM depending on whether or not the container is privileged or unprivileged. The main difference being that privileged containers allow for root access and unprived do not. it can be a large hassle to get disks or services passed back and forth to an unprived container but they are much safer to use.
Something that should be mentioned is that after initial setup proxmox is normally headless. It is controlled and viewed from a web interface. During the VM and LXC setup process, proxmox will handle the IP configuration and ports. You can tweak it if you want to, i know some people like the ip address to match the id number of the VM or container which does make identification easier. Also you can access the command line for the LXC’s but they are normally seen from whatever ip the service uses. VM’s are usually viewed through novnc so you can interact with them directly from the proxmox interface.
Why use proxmox
Why would you use something like this? I mean these are all things you could do yourself directly on any debian based installation right? Well yeah i mean you could if you wanted to go through the process of setting it all up. This is all in one place and ready to go with good documentation on how to do things. The interface alone makes the process worth it in my opinion. most of the shortcomings that i can find i have been able to work through or around as needed and redeployment in the case of catastrophic failure has been pretty easy so far.
I dont usually advocate for running someone elses scripts but proxmox helper scripts is probably the only thing that has made this even remotely useable for me. Just about all of the things that you want to run are findable there. LXC’s and VM’s both. I have done a walkthrough or two building my own LXC when there was not one on the helper scripts page but all in all it is as simple as copy, paste and follow the prompts to get things setup the way that you want. This being community run they are reviewed by others and with how proxmox is setup if one of them is causing a problem it is just a couple of clicks to nuke it.
The proxmox web interface is also a great tool for seeing how much hardware is in use. You can go to the node page and see if ram is maxed out and then go to each individual service one at a time to see which one is using what. Dont need a service anymore? You can shut it down and remove it in a few clicks
How to setup proxmox
Setting up proxmox is not the easiest thing to do correctly but i wouldnt exactly call it difficult. There are some quality of life things to keep in mind that i will mention though
Proxmox will run on just about anything, i have even seen people talk about running it on a raspberry pi so i wouldnt worry too much just grab what you have and put it to work. Grab the image from the internet and you will need to use etcher or DD or rufus in dd mode to create the usb disk. You cannot use ventoy. Next thing to remember is to have it connected to the network that you are going to use for it. WIRED network not wireless. There are some workarounds i think to make it work that way but that is something to setup later. You can get everything setup without the network connection but it is much harder to do and impossible to control until you get it working correctly. Write down the IP address. You can get it from your router but you might as well grab it now
Once the initial install is complete you can disconnect the keyboard and monitor if you want or we can put it back to use in a short time. All the rest of the setup is done from the web interface. So you bring it up from the IP that you wrote down at port 8096.
The next thing to do is go to the node since it is the only thing there and then bring up the shell for it. It is a button in the top right corner. The first thing that i always do is get the post install helper script from the proxmox ve helper script site https://community-scripts.org https://community-scripts.org/scripts/post-pve-install just copy and paste. This will update the repos including disabling the enterprise repos and will also get rid of the subscription nag. This is important to do so that you can partially automate updates.
The next thing to do is if you have another node then you need to add this to the cluster using ‘pvecm add IP’ and then do your usual update upgrade autoremove just to verify. It probably should have happened automatically but until the repos are corrected there could be an issue.
After that if you have more than one drive you need to map a directory to that drive which can then be used for VM’s and LXC’s if your primary drive is smaller or if you want a vm that uses a lot space. You can also use it to hold isos or share a portion of that drive as a mount to a VM or LXC.
Next thing to do is go out and get your iso images. The interface makes this very easy. There is a drive marked out as ‘local’ which you can go to and click on ISO images and either upload from local or you can download from URL which is basically a get command. That will make it so that you dont have to DL it locally first which can be useful. These will be the ones you are using when you do an initial install of a VM. I mean i usually at least make sure that i have the most recent image of mint. LXC’s the easiest way to get them is from the website that i posted and then to paste that into the shell for the node.
Each of these image instances has its own summary page with a notes section. I usually delete everything that is in there and put in the IP address and port in there along with any install options or set up that had to be done. the link is very useful if you can write a little html. This should not change as the container will always try to get the same IP adress. You can make it more certain by making a static ip on the router side. In most of the containers the IP is also included in summary section underneath the resources but sometimes this is not able to populate.
Next a suggestion: if you have a container or a vm that is difficult to setup or has a convoluted internal setup then i would suggest cloning it and migrating the clone. As long as you are not trying to run both at the same time you will get the same ip address even if it is move to and running on a different node in the cluster. Having the backup will save you a lot of headache later. But you probably dont need to do it for every image if they are pretty easy to replace. There are other ways to do this with things like high availability but that requires some redundancy of resources and something like proxmox backup server which requires a dedicated machine.
You can also include samba shares and nfs in the datacenter portion which will make that share available to all of the nodes in a cluster. This is great for vms but with LXC’s if you have one that is unprivileged it is very difficult to get it to be able to write to the share through the datacenter. But if you mount using a normal samba command then it is possible without doing a lot of work with users and permissioning that i was never able to get to work correctly
what i use it for and how does it help
I use my mini rack for a lot of things really. Any server service that i have set up goes there. samba, pihole, wireguard, openvpn, jellyfin, immich, audiobookshelf and a few others. I used to do this with an LXC of docker and using portainer and docker compose. That worked but the layers of permissioning got annoying and then there were issues with automated backups and if something happened that required me to get rid of the LXC then i would lose everything that in docker instead of the individual service that was the problem. A few times to get around the permissiong problems i would install a vm and then put services on that but then there was the problem that if i needed to nuke the VM then i would lose those services too plus if i had to restart the VM then everything would go down. So i do still use VM’s but for things that need a direct gui or to be able to run linux from my android phone. I also use then in conjunction with openvpn connected to my pia vpn. This means that multiple things can use a single vpn access since pia limits you to 5 connections.
I use two instances of openvpn connected to pia on two seperate machines. One of those machines i have connected to an external monitor so that i can always have a machine that is on a vpn incase i have something that requires it. This is accomplished by passing the graphics portion of the APU to the VM with pci passthrough.
The other one i use on a different machine without pci passthrough so that i can still access it using the no-vnc from a webpage which allows me to use it from my phone. On that machine i am using jellyfin which has the apu passed to it for transcoding purposes. I am using it but i need to dig further into the logs to see how it is working and make sure that it is doing everything that i want it to. jellyfin used to be setup inside a VM but that was causing me some problems when i would restart other machines and the samba shares would come unmounted. I also lost the vm when i had to do some formatting and it was too large to backup elsewhere.
I also have joplin set up there which is one of the things that i used to have setup in docker in an lxc. Joplin is a fairly simple markdown editor with some fancy setup for notes organization and server that you can install yourself so that everything is self hosted and while not real time updating on all of your devices it is pretty quick and allows me to switch between for near seamless work. I use it for basically everything that i need to write down. I am by no means an expert on markdown but i am getting the hang of it and there are plenty of cheat sheets out there.
I mentioned immich earlier and i do have that setup as well. i have a chunk of space dedicated to it on the main harddrive but i may move that to the 8tb that i have as a directory, and i also have immich pointed to a samba share. Between the two i have about 60gb of photos going back decades. Even the stuff that i converted from film to digital. This is one of the few things that i can a dedicated offsite back up for but i can talk about that another time.
I also keep an instance of resilio sync running to backup everything in my download folder on my phone and to make it so that i can very easily send things from my computer to my phone with a drag and drop. It also lets me do a bit of file automation. I can put a file into a folder on my phone and using cron and some scripting, i can do basically whatever i want.
Those three items have meant that i dont really need nextcloud right now for my own uses and they are a large part of what i was running in docker and they are all now lxc’s
One other non obviously useful LXC that i have setup is iventoy. I cannot install proxmox with it but i use it almost everytime i need to do a fresh install of an OS. It helps in that i dont need to keep track of a ventoy stick and all i need to do is make sure that netboot is enabled in the bios and make sure that it is hooked up to ethernet and i can install whatever OS i have on there. Right now i have mint fedora and tails on there along with windows 10 and 11. There are somethings that i cannot install with it but it makes reformats quick and simple in most cases. Sometimes i will need a usb to ethernet adapter but i have had zero issues with getting everything working as it should.I have even used it to load a live image to quickly access gparted to fix a disk or to use fdisk on a wonky drive.
Audiobookshelf i use daily. I have a large volume of audiobooks and i listen to them fairly constantly. I also use it for downloading the few podcasts that i still listen to but the interface for the podcasts leaves a bit to be desired when compared to something that is dedicated to it. It is a small project but i can say that it is awesome in regards to audiobooks and there is a growing following for it which is one of the reasons that it has a dedicated LXC on proxmox helper scripts page.
I also have home assistant setup in a vm since the VM provides much more robust control versus the LXC. You can do everything in the LXC that you can in the VM but you will need to do a lot more with the configuration files to get everything working. The VM just lets you do everything from the GUI.
At the center of all of this is my Open Media Vault VM which makes 4 of 8tb drives and 1 of my 4tb drives available to everything on my home network so long as you have the username and password. It is setup over samba and i know that NFS would be faster but i have yet to get that to work correctly. But for now everything is good. if i had it to do over again or if i had the spare drive space elsewhere i would have the drives in a raid configuration to increase the read write speed but it is what it is until hard drive prices come back down. But something important to mention here is the use of boot order and hookscripts. Some thing you can do with a proxmox server is choose which order things will start up in. A lot of these LXCs and VMs depend on the samba server being up and running and would either fail or would write to the incorrect locations if they started up before it is up and running. So OMV comes first in the boot order along with anything that doesnt need the drives. Then using a hookscript i connect the shares using the command line. this gets rid of a lot of the permissioning issues with passing the shares through the datacent. atleast on the main machine. Then with a slight delay to make sure that the shares are connected the openwrt instance starts which provides the vpn for certain other containers along with everything that doesnt need it and then everything else starts. I also have another hookscript on the jellyfin instance that is a pre-script instead of a post-script that reconnects the samba shares on another machine. that way if the share was bounced i dont need to bring everything back down and up to get it all connected again. just bounce the jellyfin instance.
I dont know if you can tell from what i have said so far but i use these machines for a lot of things and a lot of testing of new things. One of the important aspects that i have mentioned is the ability to bring a single container down and back up because each is running independent of the other. If one has a memory issue it is isolated and i do not need to bring down the whole server to fix it. I can start up a new container very quickly and test whatever it is i am wanting to test and then destroy it after. If something needs a reboot because of an update then once again its just that one container that needs to be down. I can take a back up of a container with a couple of clicks and then do some testing on the original and then if i mess it up i can just nuke that container and bring up the backup. Could you do all of these things from regular operating system? Sure but the interface makes it super easy and adjusting the resources to match the task is brilliant. Plus with most computers, how often are you using the whole thing? Unless you are a gamer or video editor i would say not that often. But now i can put those unused cycles to use for various things and determine how much headroom i have left to do others.
Now are there other ways to do this? Yes, my understanding is that you can do very similar with truenas or OMV and a couple of other dedicated hypervisor operating systems. I have also heard some people say that they are better solutions for whatever reason. The most prominent that i have seen being that proxmox itself has some built in overhead and a high level of complexity.
I have probably gone on too long but hey this is my current hobby. Next year ill probably be juggling geese or something.
