This Week in Self-Hosted (1 December 2023)
This week's newsletter features a ton of notable software updates, a content spotlight on #AudioBookshelf, video content from @dbtechyt and Jim's Garage, and a new edition of the Open Home newsletter from @balloob!
There are times when I want to re-do my #homelab so it looks more professional. It's just a pile of computers right now.
It's usually the sign of a good day when a miscellaneous cat6a matches the flowers on my dress. 💁🏼♀️
Secondary to ethernet distractions, it's definitely a good day when two additional #Nvidia #Tesla T40 datacenter GPUs arrive for the #homelab! Now the question becomes whether to install both immediately, or wait until another is sourced so that each of the new #IBM #POWER9 systems will deploy with the same spec. Decisions... 🧠
But before I continue with that, I'm going to spend some time thinking about the migration itself, just to get myself motivated. I want to be ready to go into full swing migrating my Nomad cluster to Kubernetes during my Christmas vacation.
The overarching plan is to keep the entire Homelab up throughout the migration. I'm pretty confident that I have enough hardware, the only thing really missing now is good plan.
After somewhat accidentally fixing my netboot problem, it is now back to the k8s migration. I think I've had enough of experimentation and I'm pretty confident that k8s is at least fit for purpose. There's one task left I have to do though, namely testing a Pi 4 first as a control node and then as a worker node.
#MediaWiki #Update: 🙈 Erstmal #Ubuntu 18.04 im #LXC auf 20.04 updaten, damit zumindest #PHP 7.4 verfügbar war. Danach MediaWiki selbst von 1.31 auf 1.35 (#LTS) und nochmal auf 1.39 (LTS) hochgezogen. Letzter Schritt: Das ganze rüber bringen auf #AlmaLinux 9 mit PHP 8.2. 🚀
#Homelab #LinuxContainer #VM #VirtualServer #SelfHosted #Linux #RHEL #Proxmox #Cluster #Server
Achei praticamente uma "Revistinha Avon" de softwares opensource para self-hosting
@firstname.lastname@example.org Here’s the updated #homelab! I mounted and ran the cables for the Pi Zero 2 W and it’s running Pi-Hole. I also swapped out the white zip ties on the black cables for black ones so everything is color matched. I think it’s a much cleaner look now. All that’s left is getting longer network cables for the floor-mounted machines and running those up to the switch.
It pays to be your own cloud provider #homelab https://lifehacker.com/tech/dont-rely-only-on-cloud-storage
I shuffled half of my rack around just to get my Aggregation Switch in the right place, 1u at a time. 😅
The follow-up post to my guide how to build a NAS from an old micro PC is finally published, explaining how to get up and running with TrueNAS Scale, gotchas I came across and my favourite apps:
In the past week, I have migrated my desktop virtualization servers running VMware ESXi to Proxmox VE. I wrote up a post about it and some of my thoughts.
Managed to process the correct amount of downloads for @buttercup desktop using #n8n. Just needed some proper filtering of non-binary assets from #github. Now that I have this and the browser extension stats, it might be time to write this into a database of some kind.. (or just transfer it via #mqtt)
Critical Vulnerabilities in common #homelab software is why I keep everything in mine behind a well maintained VPN. Don't expose your personal lab to the internet, even companies with full security teams fall victims to vulnerable software. It's unlikely you'd notice something in your lab until it's too late. https://arstechnica.com/security/2023/11/owncloud-vulnerability-with-a-maximum-10-severity-rating-comes-under-mass-exploitation/
A quick post about setting up a VM to work on netbooting scripts. Also contains some frustration with VirtualBox and a couple of diagrams since I rediscovered Excalidraw.
BUT there is another log, from the second node, which executed the backup job for this migrated VM.
So all fine...
Learned alot already about setting up this tiny cluster. This is exactly what I hoped for.
So, Google is now calling it "sync issues".
Anybody want to bet how long it is until OneDrive also suffers "sync issues"?
But the crux of the matter is: I've never had "Sync issues" (read: Actual lost files) with any of the more or less janky solutions I've run in my homelab for file sync over more than a decade. I didn't even have data loss in the times before I started having backups.
This timeline is going from weird to freaky.
'Don't worry about this, there was a misunderstanding. A server will not be housed in an employees domicile.'
I have never been more concerned over an email in my life than I am right now. Like, good. I'm glad its not happening. But why the hell was it going to?? Did the developers think 'homelab' meant they get to take the servers to their home??
And that remote monitoring is live. Much smarterer of me to do that outside of my network instead of in so I can ya know. Know when things lose power.
First impressions on #Jellyfin: it's better than I would have expected, but the metadata agents leave something to be desired with mismatches and downloading incorrect media artwork. I'm also not wild about user management. Still, I'm willing to give my users options given how #Plex has been continuing to do wrong by users over the years. #homelab
Wow did Plex shoot themselves in the foot. What the hell were they thinking not making that sort of data disclosure opt-in? What did they expect would happen?
I must also admit that I had no idea how much Plex was "selfhosted" in quotes. I always had it pegged for something like Jellyfin, but with a Fremium model or something. I never realized that they were embedded deeply enough into your "selfhosted" setup to be able to freely distribute your viewing habits.
Out with the old, in with the... slightly less old but MUCH faster!
I've upgraded my ageing Lenovo M93p Tiny for use as a low-power Haiku/VM host/compiler box, going from an i5-4570T to an AliExpress-special Xeon E3-1275L V3.
With the hyperthreading and 8MB of L3 cache, PassMark says that this new processor is almost double the speed of the old one. But it only increases the TDP from 35W to 45W, so no need for a bigger PSU.
Okay, the homelab needs some work.
Specifically, I want to shift from a single Linux VM hosting ~10 docker containers (with various shared networks between them) to a kubes deploy.
1. I don't know how feasible it is to go from docker-compose ==> k8s
2. I have never stood up a k8s deployment before, I'm thinking a one master, 2 worker node layout? 3?
3. My cursory look at k8s seems like there isn't "one" way to do things (fine), so where does one start to learn/deploy?
26 hours into some badblocks testing… thank goodness for tmux, and the endless patience of computers.
I'm dreaming something like this:
- height ≤ 1U, with optional rackmounting ears
- width ½ or ⅓ 19″ so 2 or 3 could fit in a rack side-by-side
- depth: ≤ ~30cm
- 2 × NICs
- 2 × NVMe 2280 M.2 storage
- ≥ 64GB RAM
- 2 × USB A
- Nice to have: 2 × USB4 Type-C
- Don't care about video and audio I/O
- Don't care about WiFi and Bluetooth
- Front I/O
- Backside C14 power
- FtB airflow
A bit'o #homelab history. I just found an invoice showing my purchase of a 320 GB Western Digital EIDE drive for the princely sum of $77.23
You have to be fucking kidding me. For months, at least two of my nine netbooting hosts had problems coming up after a reboot in every single update run for the Homelab hosts.
But now. Now that I have some logging for my headless, netbooting machine's initramfs boot step setup - everything works flawlessly.
If it wasn't Sunday evening, I would be getting out the Lagavulin now. 😒
Yak shaving completed! 🎉
All of my netbooting hosts now send all console output to netconsole, which sends them to my FluentD instance to be stored in my Loki. It should now get all error logs even from headless hosts during the initramfs phase.
Now onto the Homelab hosts update. The one thing I need now is for at least one of my hosts to run into the reboot problem and producing useful logs which finally tell me what the problem is.
Had a problem with my #Terraform setup when I updated to #Proxmox 8.1. The provider I've been using, telmate/proxmox, isn't getting updates anymore. I found @bpg's provider, which is still being worked on, and I'm switching to it.
Everything seems to be going well! #homelab
I'll be updating the #homelab pegboard in the coming days as I mount one of my new Raspberry Pi Zero 2 W units that I impulsively picked up. This particular one will be taking over as the Pi-Hole for my network, which is currently run by an LXC container on my main NUC server. I have chosen to migrate to the Pi because I am strapped for resources on the NUC and setting up the Zero 2 W was cheaper than buying a DDR5 SODIMM upgrade kit.
My find from the #IndieAppSales is Pi-Hole Remote. A bargain at the $1.99 price, which covers iOS, MacOS, and even WatchOS devices. And..you can monitor multiple PiHoles and sync configurations between them.
And check out the full set yourself - over 400 apps included in the sales today. Support your Indie Devs!
PSA: If you upgrade your #Proxmox machines to 8.1, the #Terraform provider telmate/proxmox will stop working. It's a known issue and the project seems orphaned: https://github.com/Telmate/terraform-provider-proxmox/issues/863
Another good alternative seems to be: https://registry.terraform.io/providers/bpg/proxmox
- LXC templates
- Defining network interfaces for Proxmox
- Passthrough in #Proxmox for grub and systemd-boot
- Minecraft Proxy using Waterfall
- Minecraft Worlds using PaperMC
#Proxmox and the network adapter in my mini PC aren’t really friends.
I upgraded to 8.1 and I had to plug in monitor and keyboard to get connectivity back :(
It’s fixed now but I really wish I had got a vPro system.
On the flip side, the in-tree kernel module works well after the software update and hopefully from now on I won’t have any more issues.
Ha! Ordering these same components I just recognized that my existing server and with it this #Mastodon instance is running since a little bit more than a year already!
And of course it’s all based on excellent people 😊
I know I been a big bbbiaaatch about this but here is a question. In your storage... whatever ..would you take ..
1. More bays to fill ya up with PB's of data... ( I know it sounds sus just go with it )
2. Go Fast and Furious 43! 10 gig on dem hoes.
Some context. I have 10 gig available to me and realistically I don't need Petabytes of space...
Sometimes, when I read those "Twitter is better than Mastodon" posts from larger accounts who are announcing that they're going to spend less time here, I'm reminded how bubbly the Fediverse really is.
Reading other people's "quasi-goodbye" posts, it looks like the Fediverse is an absolute cesspit of scolds and reply guys.
I can only come to one conclusion: The Homelab bubble on here must be a bunch of exceptionally nice people, because I never got any of that.
I also learned I can run a playbook on the newly created resource right from Terraform. From what I've read it's not the best practice, but at such a small scale it doesn't really matter much.
Black Friday ads have my number. Lots of sales on mini pcs for a home lab. Must. Resist.
@matt Have you ever looked at the #MiniPCs that have laptop chips inside? I have two of those and frankly, they are amazingly capable. I don’t think most people realize they exist or what the benefits are: low cost, VERY low power draw, flexible enough to run #Windows or #Linux very easily.
Some thoughts after buying my first one, which I’ve now relegated to a #homelab server and bought a more powerful one for daily use: https://www.kctofel.com/so-my-new-desktop-pc-was-delivered-in-a-padded-envelope/
Bet you could get a review unit lickety split!
One of these days I'm gonna reorganize my #homelab setup so that I don't just have all the configs inside my home folder, I promise :blobglarenervous:
The home lab now has some pretty dashboards thanks to Grafana. This was my most difficult setup so far, and my first docker stack with more than one app installed in it.
Ran into issues with the stack crashing a LOT in portainer while I fiddled with it. :yikes: Not sure I made the right choice as to the appropriate server to install it.
Back in my tech support days I thought that if I had PXE network boot at home, that I "made it". Well, that day has come! This past week I learned all about netboot xyz! I can now boot and install any operating system over the network!
Check it out! https://www.youtube.com/watch?v=4btW5x_clpg
Today, I setup a multi-step backup process for my day job that I would have never have understood if it weren't for my home lab experiments. Felt good.
Keep having fun, keep learning!
And, if you do opt for the TrueNAS XL+, be warned that the thermal design of the chassis is *not* that optimal for the top set of drive bays.
I also like that you can order the HL15 sans motherboard, CPU and RAM for $910. That way, you get all of the backplanes, cables and power supply installed, and you can just source the motherboard, CPU, RAM and storage controllers of your choice.
That way, you can use a Ryzen, Threadripper, EPYC, Xeon D, Sapphire Rapids or even an Ampere setup in there (granted, you'll want to upgrade the power supply)
I would *love* to see someone plop in an Ampere Altra board into an HL15 system. Maybe that can be a future project for @geerlingguy? 🙃
Watching @CraftComputing's video on the 45HomeLab HL15 server. For around $2200 for a full-build server with 32 GB of RAM that can support 15x 3.5" hard drives is a lot better than the TrueNAS Mini XL+ 8x 3.5" hard drives with 64 GB of RAM at about $3000.
If the HL15 were out when I was building up my new NAS, I probably would have gone with that over the TrueNAS X+ that I picked up. The X+ configuration has 32 GB of RAM, same as the HL15.
The HL15 has fewer CPU cores (6) vs the X+/XL+'s 8 cores, but the X+/XL+ uses Intel C3000-series Atom cores versus full Xeon cores. Plus, you can stuff the HL15 with a lot more RAM, network cards and other PCIe cards. The tradeoff is size and noise, of course.
It's a bit silly to see the the living room rack become more populated and dense, considering one of my budget rationalizations for the three colocation racks was "The cost difference will offset electric use at home and it'll be so much quieter...", yeah sure. Well, those racks are at circuit capacity so... 👩🏼💻🤦🏼♀️💁🏼♀️
I would *absolutely* be interested in knowing how other folks solve this #homelab challenge.
Power goes out, your UPS gives out after a period of time...then everything shuts down, requiring a power button press to bring it all back up!
No, I'm not moving this stuff to a cloud provider or data center. 😉
My server boards do not have IPMI. Would that allow me to boot them remotely once power returns without pressing a button?
Any other solutions?