Well hello there, it has been a while since we lasted posted some homelab updates. Rest assured we have not forgotten about the blog, neither has it been for a lack of doing. Between work, life and odd jobs, it really has been a busy few months. In the meantime, this update covers what we have been busy with and hope to achieve over the coming months, so stay posted!

Update from James

Over the past year I have dialed in my homelab, and after a wealth of changes at the start of 2023, my homelab has been left mostly unchanged. It’s serving its purpose, providing a place to run and host various applications like Nextcloud, Plex, Immich, and so on.

These past few months I have been busy focusing on my personal growth in software development. For those unaware, I am a software developer by day, and the past year I really fell in love with Golang. It is a simple, yet powerful, opinionated language perfect for web applications, CLIs, and more.

Over the last few months I have built out a small set of utility CLIs for managing, matching and processing the media I rip from my collection of discs. I plan on putting together a piece about developing your own tools for the homelab when existing tools either don’t fulfil your needs or exist in the fist place.

In the coming months, I plan to document and build out my own self-hosted CI/CD pipeline in the homelab using Gitea. The idea being it can run checks on code I commit, build binaries for my CLIs, and deploy web applications to a self-hosted container registry, before then deploying to my server.

While I have done this professionally with GitHub, Azure and GCP, it would only be part of The Homelabber ethos to self-host my own system!

Update from Blag

This past winter felt a bit like a hibernation period as I worked to create a better work-life balance and find better times of the day to work on projects and writing them up.

I have been spending a lot of time honing and improving hosted services in terms of quality and making maintenance easier. My homelab is a means to fulfilling my miniature data centre ambitions. Spinning up new services is relatively easy, building stability and maturity are the real challenges.

Monitoring and alerting have been crucial in quantifying performance and making improvements, all of which has been feeding into what is now the third iteration of the monitoring article I have been working on since last year.

I did also find time to pick up Hugo and build a portfolio site for my sister. There is some pretty cool work coming up around pairing up self-hosted mail campaign tools with external email delivery platforms to digitize event and concert invitations.

Besides wrapping up on current projects, my goal is to free up time in June to focus on learning Ansible and Python.

Self-hosted Offsite Backups

James and Blag are in the early stages of a joint project with the aim of hosting secure and encrypted offsite backups for one another. Both of us have grown tired with either the limitations or occasional issues associated with our current offsite backup solutions.

We are thinking of using Resilio as it offers simple and encrypted syncing. But if you already do this, or know of a better tool, let us know!

Storage Upgrades

The moment we all knew was coming is here, needing more disk space.

Dedicated Web Service Storage

Creating a new RAID 1 SATA SSD storage pool for use with some production workloads. This was born out of a desire to decouple the storage requirements for services like Nextcloud and downloadable products on one of my sites.

Resizing the Unraid Array

Both James and I upgraded and increased the size of our Unraid arrays and resized them to larger capacity parity drives. This was a first for both us in Unraid and some complications were bound to happen.

James’ Server

I use my old Synology NAS for a local backup of my Unraid server, however it is only a 2-bay containing two 6TB 5400RPM disks. With a growing media library, this simply wasn’t cutting it anymore. I initially ordered a new 4-bay Synology DS423 (non-plus), only to learn the arm CPU is not supported by some of the Synology software I use. And so I ended up with four DS423+. Yes you read that right, I was accidentally sent 4… of which 3 were returned.

The DS423+ uses an x86 Intel CPU, which just so happens be the same one as the DS720+ it was replacing.

My plan was to pull the two 6TB disks from my Unraid server (including the supposedly failing disk), adding them to the Synology, replacing them with a new 16TB disk.

This required adding the new 16TB disk to Unraid first, however my parity disk for the two 6TB disks was a 12TB disk… This required me to do a bit of parity juggling, making the new 16TB disk a parity and then using the previous 12TB disk in the storage array instead.

Between the parity juggling and long rsync commands, this migration was a bit of a nightmare.

Blag’s servers

I originally sized my personal storage servers with 4TB drives to take advantage of my existing drives and the pricing of 4TB disks back in 2021 was still attractive. My plan was to run with this array configuration until the time came to start swapping out older disks.

I have a second server that runs all my production workloads including publicly facing websites, databases, backups from the primary server, and Traefik and Nginx proxies. Its array started life with 16TB and 4TB disks running without a parity. I eventually added another 16TB drive as a parity disk. The intention is to at a minimum for both servers to have the same storage capacity.

The last time I bought new disks I ended up with two being dead on arrival, since then I always bench test my disks.

Jumping forward to April 2024 the time to upgrade disks has finally come and proceeded with the following plan.

• Upgrade the parity disks to 16TB.
• Introduce a 16TB data disk to increase the pool size.
• Remove and RMA a failing disk.
• Replace my oldest disk with the replacement RMA disk.
• Take the remaining 4TB disk and add it to the secondary server.

I used the Unbalanced plugin to move files off the disks that were being replaced. This is also a great plugin for redistributing files for more even space consumption.

This is not the first time Scrutiny has flagged a failing drive, this time with the dreaded reallocated sector count. I have since received a replacement unit from Seagate and is pending a bench test before adding it to the array. I plan on replacing my oldest drive, an eight-year-old 4TB HGST Deskstar. They sure don’t build em’ like they used to.

An important lesson I learned was to pay extra attention to the disk serial numbers and not solely rely on the physical labels I had put on the drives. My biggest challenge and self-inflicted snaffu was moving the wrong disk over to the hosting server, which made the other array very unhappy. A quick server pull and disk swap later all was good in the world and ready for a parity sync. Fast forward to nearly four days later worth of parity syncs and disk pre-clearing and I now have a 32TB array!

Preparing for Home Automation

Blag finally took the plunge into RGB room lighting with the purchase of Govee lights and IKEA TRÅDFRI bulbs. He also recently picked up a Midea portable air conditioning unit that should play nicely with Home Assistant.

My plan is to make Home Assistant the core of the automation setup and use Scrypted to fill in the gaps. The latter is just an assumption as I have not started working on this project yet.

Integrating my CCTV feeds into the home automation setup will be the final piece. Logging into a separate application for my CCTV feed is not a huge issue.

Fresh Shinobi CCTV

I have been using Shinobi as my CCTV solution for two years now, but have been using an older container from Spaceinvader One that has not received updates in quite some time.

I have switched to the official Shinobi project and images and have built a fresh config. There will be content around this, specifically getting the GPU TensorFlow plugins to work and filling in gaps in the documentation.

This is all still in testing until I get all the cameras into their final positions along with the ethernet cabling.

Office Unraid Server

Blag built a new server at a client site to replace a NAS that had reached end of life and provide enough compute resources for future projects. I shot a timelapse video fo the build, we will see what comes of it.

There is a write-up in the works and no build is complete without drama and troubleshooting.

I have deployed a small Prometheus and Grafana based monitoring stack to keep tabs. I also have to say I am quite pleased with Unraid’s Wireguard VPN implementation, it has proven robust and the fact that service runs independent of the array has made it invaluable for troublshooting and maintenance.

The next step for this server will be to provide an email archive using Mailpiler in order to import over a decade worth of emails and connect it with various mailboxes for continued synchronization.