Edit2: OK Per feedback I am going to have a dedicated external NAS and a separate homeserver. The NAS will probably run TrueNAS. The homeserver will use an immutable os like fedora silverblue. I am doing a dedicated NAS because it can be good at doing one thing - serving files and making backups. Then my homeserver can be good at doing whatever I want it to do without accidentally torching my data.
I haven’t found any good information on which distro to use for the NAS I am building. Sure, there are a few out there. But as far as I can tell, none are immutable and that seems to be the new thing for long term durability.
Edit: One requirement is it will run a media server with hardware transcoding. I’m not quite sure if I can containerize jellyfin and still easily hardware transcode without a more expensive processor that supports hyper-v.
Virtual machines also exist. I once got bit by a proxmox upgrade, so I built a proxmox vm on that proxmox host, mirroring my physical setup, that ran a debian vm inside of the paravirtualized proxmox instance. They were set to canary upgrade a day before my bare-metal host. If the canary debian vm didn’t ping back to my update script, the script would exit and email me letting me know that something was about to break in the real upgrade process. Since then, even though I’m no longer using proxmox, basically all my infrastructure mirrors the same philosophy. All of my containers/pods/workflows canary build and test themselves before upgrading the real ones I use in my homelab “production”. You don’t always need a second physical copy of hardware to have an appropriate testing/canary system.
I really like this strategy. I currently use proxmox for my home server needs, but I am curious what you use now instead?
I have condensed almost all of my workflows into pure bash scripts that will run on anything from bare metal to a vm to a docker container (to set up and/or run an environment). My dockerfiles mostly just run bash scripts to set up environments, and then run functions within the same bash scripts to do whatever things they need to do. That process is automated by the bash scripts that built my main host. For the very few workflows I have that aren’t quite as appropriate for straight docker (wireguard for example) I use libvirt to automate building and running virtual machines as if they were ephemeral containers. Once the abstraction between container and vm is standardized in bash, the automation doesn’t really need to care which is which, it just calls start/stop functions that change based on what the underlying tech is. Because of that, I can have the canary system build and run containers/vms in a sandbox, run unit tests, and return whether or not they passed. It does that via cron once a week and then supplants all the running containers with the canary versions once unit tests pass.
Basically I got sick of reinventing the wheel every time a new technology came out and eventually boiled everything down into bash so that it’ll run on anything it needs to. Maybe podman in userland becomes the new hotness next year, or maybe I run a full fat k8s like I do at work. Pure bash lets me have control over everything, see how everything goes together, and make minor modifications to accommodate anything I need it to.
It sounds more complicated than it really is, It took me like a week of evenings to write and it’s worked flawlessly for almost a year now. I also really really really hate clicking things by hand lol, so I automate anything I can. Since switching off proxmox, this is the first environment that I have entirely automated from bare-metal to fully running in a single command.
I’m incredibly lazy; it’s one of my best qualities.
That sounds pretty slick. I envy your scripting prowess. You really have to know your system top to bottom to be able to boil it all down like that.
I’m just beginning my journey into this whole space, and it’s really interesting how many different ways people have to deal with the same basic things.
I’m also incredibly lazy, so maybe more scripting is in my future! Thanks for taking the time to write such a detailed reply!
I certainly wasn’t just born good at this. Unironically if you want to learn how something works, try to automate it. By the time it’s automated you’ll understand basically every part of it at at least a basic high-level.
That is incredibly true. I try to automate everything I can. That’s where laziness is a superpower.