diff --git a/src/_posts/infra-snapshot-2024/2025-01-12-infra-snapshot-2024.md b/src/_posts/infra-snapshot-2024/2025-01-12-infra-snapshot-2024.md new file mode 100644 index 0000000..02977d5 --- /dev/null +++ b/src/_posts/infra-snapshot-2024/2025-01-12-infra-snapshot-2024.md @@ -0,0 +1,245 @@ +--- +layout: post +title: Home Lab Infrastructure Snapshot 2024 +date: 2025-01-12 20:49:00 Europe/Amsterdam +categories: infrastructure homelab +--- + +If you are reading this when this post was just published: fear not, you did not +just travel back in time to the glorious old days of '24. No, I was simply too +~~lazy~~ busy to write this post last year. Busy with what? Well, I basically +changed every part of my infrastructure the past year obviously. This is all due +to the fact I got infected by the NixOS-virus! + +This post is a follow-up of a similar post from 2023 where I described my whole +home lab as it was back then. In this post I will mainly focus on the things +that changed compared to then. + +# Hardware and Operating Systems + +## Hardware + +It seems I was rather frugal this year, as I did not buy any new servers or +replaced existing ones. + +However, as we will see in a bit, I am now running Linux on my gaming PC and +deploying it just like I would do with any one of my servers. Therefore, I will +take a moment to gloss over the hardware of that machine: + +- **Motherboard**: Gigabyte B550 GAMING X V2 +- **CPU**: AMD Ryzen 5 5600 +- **GPU**: AMD Radeon RX 6700 XT + +Here is a picture of its guts: + +![The inside of my gaming PC showing its components.](gamepc.jpeg) + +You probably noticed the random components inside it, which are fixed just +securely enough to not cause a fire. These are actually parts of a +[DIY PiKVM V2](https://docs.pikvm.org/v2/). PiKVM is a cool project to transform +your Raspberry Pi into a KVM (Keyboard, Video and Mouse). A KVM, in its most +basic form, allows you to see the video output and control the mouse and +keyboard of a system remotely. + +In the bottom right part of the picture, connected to the ribbon cable, you can +see the part that allows the Raspberry Pi to power on/off the PC, reset the PC, +read power status and read disk activity, all via its GPIO! In the top right +part of the picture you can see the component that converts the PC's HDMI output +to CSI, which the Pi supports. Just behind that component, but out of view, the +Pi is secured. + +PiKVM has some more neat features: it can emulate a USB storage device which I +can boot off of. This allows me, for example, to run a memory test or tweak BIOS +settings. + +One other piece of hardware is worth mentioning. Similar to the PiKVM, I also +bought a +[Sipeed NanoKVM](https://wiki.sipeed.com/hardware/en/kvm/NanoKVM/introduction.html). +I opted for the Lite variant, which is only able to read HDMI and interface with +the hosts USB. It can't control the power like the PiKVM that I built. For now I +attached it to one of my Kubernetes nodes. + +![Unboxing of the NanoKVM lite.](nanokvm.png) _Unboxing of the NanoKVM Lite, +[courtesy of Sipeed](https://github.com/sipeed/sipeed_wiki/blob/main/docs/hardware/assets/NanoKVM/unbox/lite_ubox.png)._ + +## Operating Systems (or, the NixOS propaganda) + +For people that know me in real life, this section will not be a surprise. In +late 2023 I started getting interested in NixOS and in 2024 I worked hard to +install it on any piece of hardware I could get my hands on. + +NixOS enables me to apply my Infrastrucutre-as-Code spirit to the OS and take it +to the extreme. I used to customize an existing OS (mainly Debian) with Ansible +to configure it to my liking. This works to some extent: the steps needed to get +to my desired configuration are codified! However, after using Ansible for some +time I started seeing drawbacks. + +Ansible likes to advertise itself as declarative (because it's YAML, which is +declarative by nature right?). Sure, all steps _should_ be declarative and +idempotent (they can be run repeatedly without changes). But when you do +anything non-trivial, you start to introduce dependencies between different +tasks. Suddenly, your tasks cannot be run just on their own anymore! Together, a +certain Ansible role should indeed still be idempotent, but I often have to +think about the current state of my machines still. + +In contrast, NixOS is configured entirely declaratively. You don't have to tell +NixOS how to go from configuration A to configuration B, this is something NixOS +will figure out for you. Not convinced? This is how you get a fully functional +Kubernetes node on NixOS: + +```nix +{ + config.services.k3s.enable = true; +} +``` + +### Colmena + +Deploying changes on a single NixOS host works fine with the `nixos-rebuild` +tool. However, if you manage multiple NixOS servers, a bit more comfort is +quickly desired. That's why I am happily using the +[Colmena](https://github.com/zhaofengli/colmena) deployment tool. Below is an +example deployment for my servers. + +```shell +❯ colmena apply --experimental-flake-eval --on @server +[INFO ] Using flake: git+file:///home/pim/git/nixos-configs +[WARN ] Using direct flake evaluation (experimental) +[INFO ] Enumerating nodes... +[INFO ] Selected 4 out of 6 hosts. + ✅ 2m All done! + (...) ✅ 37s Evaluated warwick, atlas, lewis, and jefke + atlas ✅ 0s Built "/nix/store/r0jpg2nrqdnk9gywxvzrh2n3lrwhzy56-nixos-system-atlas-24.11pre-git" +warwick ✅ 4s Built "/nix/store/kfbm5c2fqd2xv6lasvb2nhc8g815hl79-nixos-system-warwick-24.11pre-git" on target node + lewis ✅ 0s Built "/nix/store/h238ly237srjil0fdxzrj29ib6blcmlg-nixos-system-lewis-24.11pre-git" + jefke ✅ 0s Built "/nix/store/b7pnan3wmgk3y0193rka95i82sl33xpc-nixos-system-jefke-24.11pre-git" + atlas ✅ 10s Pushed system closure + jefke ✅ 5s Pushed system closure + lewis ✅ 9s Pushed system closure +warwick ✅ 53s Uploaded keys (pre-activation) + jefke ✅ 49s Uploaded keys (pre-activation) + lewis ✅ 47s Uploaded keys (pre-activation) + atlas ✅ 46s Uploaded keys (pre-activation) + jefke ✅ 9s Activation successful + atlas ✅ 18s Activation successful +warwick ✅ 4s Activation successful + lewis ✅ 16s Activation successful +``` + +Notice the use of `@server`, which only deploys hosts thare are tagged with +`server`. I also deploy my laptop and gaming PC with this tool, so this is very +handy. Also notice that the host `warwick` is built on the target node. This is +because `warwick` is a Raspberry Pi 4 with an ARM architecture. The system is +therefore compiled on the remote. + +## Virtualization + +I had two main purposes for virtual machines: security and host network +isolation. I used to run Docker Swarm on my servers, which messes around with +your `iptables` rules. Virtual machines were therefore a way separate that from +the main host. + +However, I found that VMs are pretty difficult to manage. It's difficult to +define them in an infrastructure-as-code way. To this end, I used Terraform with +the pretty buggy +[terraform-provider-libvirt](https://github.com/dmacvicar/terraform-provider-libvirt) +provider, but this was not a smooth experience. I also found them running out of +memory quite often. And while I could have invested time into something like +memory ballooning, I didn't really want to spend my time with that. + +So now my servers are completely VM-free! + +## Container Orchestration + +Alongside the operating system, my container clustering setup changed the most +this year. + +Before this year, I was using +[Docker Swarm](https://docs.docker.com/engine/swarm/) for container clustering. +The main benefit for me, was its similiarities with Docker Compose, which I was +using at the time. Unfortunately, Docker Swarm is not widely used and doesn't +seem well maintained either. Also, a feature I was really missing was the option +for "distributed" storage that syncs data between nodes to mitigate hardware +failure. + +With Docker Swarm out of the option, I needed to choose another solution. +Initially, I wanted to use [Hashicorp Nomad](https://www.nomadproject.io/). +Unfortunately, Nomad is no longer open source software, so this is out of the +question for me. Then, apart from some smaller projects, I really only had one +option left: **Kubernetes**! + +Below I outline some of the components I use in my Kubernetes setup that make it +tick. + +### k3s + +I opted for the [k3s](https://k3s.io/) Kubernetes "distribution", because I +wanted to start simple and as you saw in +[a previous section](#operating-systems-or-the-nixos-propaganda), it's super +simple to enable on NixOS. k3s has all the Kubernetes components out-of-the-box +to run a single-node cluster. + +In the future, I might swap to using NixOS' `services.kubernetes` +[options](https://search.nixos.org/options?channel=24.11&from=0&size=50&sort=relevance&type=packages&query=services.kubernetes) +as I hear these work great as well and gives you more control over what runs. + +### MetalLB + +If you use the Kubernetes platform of a cloud provider, like +[Microsoft's AKS](https://learn.microsoft.com/en-us/azure/aks/), +[Google's GKE](https://cloud.google.com/kubernetes-engine) or +[Amazon's EKS](https://aws.amazon.com/eks/) (all really inspiring names, +probably great to convince your upper management), they provide network load +balancers by default. These are great, because they simplify exposing services +outside of the cluster. Unfortunately, bare-metal Kubernetes lacks a load +balancer and this is where [MetalLB](https://metallb.io/) comes in. + +MetalLB works by assigning locally-routable IP addresses to Kubernetes services. +It has two methods to do this: either via ARP or via BGP. I opted for ARP to not +overcomplicate my network setup. To use ARP with MetalLB, you simply have to +reserve some IP address space for MetalLB to play with. It will dynamically +assign these IPs to Kubernetes services and advertise these via ARP. + +### Longhorn + +Similar to MetalLB, [Longhorn](https://longhorn.io/) fills a gap that Kubernetes +has on bare-metal deployments. Without Longhorn, any Kubernetes storage volume +is tied to a particular host. This is very problematic when you want to move +containers around, or if a physical server dies. + +Longhorn fixes this by replicating block storage across multiple Kubernetes +nodes. Want to take down a node? No problem, the workloads can move to another +node with the same data. Longhorn can also create periodic backups, which I +personally backup off-site to [BorgBase](https://www.borgbase.com/). + +### Cert-manager + +Another amazing part of my Kubernetes setup is +[Cert-manager](https://cert-manager.io/). Cert-manager automatically manages TLS +certificates that are needed for your Kubernetes deployments. + +Usage of Cert-manager is super simple. First you have to set up a certificate +issuer, like Let's Encrypt. Then you can request Cert-manager to automatically +provision certificates. I use Kubernetes ingresses, and can simply give it the +`cert-manager.io/cluster-issuer` annotation. Then, Cert-manager will use the +`host` field to request the certificate. + +I abstracted this a bit, and this is for example the ingress definition for the +blog you are reading right now: + +```nix +{ + lab.ingresses.blog = { + host = "pim.kun.is"; + + service = { + name = "blog"; + portName = "web"; + }; + }; +} +``` + +### Tailscale + +## Workloads diff --git a/src/_posts/infra-snapshot-2024/gamepc.jpeg b/src/_posts/infra-snapshot-2024/gamepc.jpeg new file mode 100644 index 0000000..d5e7824 Binary files /dev/null and b/src/_posts/infra-snapshot-2024/gamepc.jpeg differ diff --git a/src/_posts/infra-snapshot-2024/nanokvm.png b/src/_posts/infra-snapshot-2024/nanokvm.png new file mode 100644 index 0000000..82499e2 Binary files /dev/null and b/src/_posts/infra-snapshot-2024/nanokvm.png differ diff --git a/src/_posts/parking-expenses-automation/2025-03-02-parking-expenses-automation.md b/src/_posts/parking-expenses-automation/2025-03-02-parking-expenses-automation.md deleted file mode 100644 index b450921..0000000 --- a/src/_posts/parking-expenses-automation/2025-03-02-parking-expenses-automation.md +++ /dev/null @@ -1,102 +0,0 @@ ---- -layout: post -title: Automating a Parking Expenses Report -date: 2025-03-02 14:38:00 Europe/Amsterdam -categories: paperless paperless-ngx nix python python3 ---- - -For the past year, I was consulting at a place where I could request -reimbursement of parking expenses. This had to be claimed using a standard Excel -template. - -Doing this manually every month seemed boring, so I automated a significant part -of that: - -- Fetch parking costs for a certain month from my Paperless-ngx instance -- Calculate costs after taxes and a grand total -- Fill in an Excel sheet with the details -- Convert the Excel sheet to PDF and append photos of parking costs - -Note: I don't expect anybody to have the same requirements as me here, but -hopefully pieces can be useful to some. The open-sourced project can be found -[here](https://git.kun.is/pim/parking-expenses). - -# Organizing the Parking Costs - -Each time I would park, I would accumulate a parking ticket. This ticket -contains the following information: - -- Date and time -- Car license plate -- Price -- VAT - -In order to automate the report, I had to have this information in a structured -way. Therefore, I turned to [Paperless-ngx](https://docs.paperless-ngx.com/). -Paperless-ngx is an self-hostable open-source service that helps digitalizing -paper documents. Third-party mobile apps help quickly scanning documents and -uploading them to the server. This is exactly what I did with the parking -tickets as well: each time I received a ticket, I would scan it with the -Paperless app. - -Paperless-ngx uses Optical Character Recognition (OCR) to cleverly extract the -date of the ticket automatically. I also created a custom label to indicate a -parking ticket, and I added custom fields to indicate a ticket's cost and VAT -amount. You can see this information in the screenshot below. - -![foo](ticket.png) _Translated from Dutch: aanmaakdatum means creation date, -parkeerkaart means parking ticket, BTW means VAT and bedrag means cost_ - -Paperless-ngx will also automatically learn to label the tickets with the -"parking ticket" label, based on the document's content. That just leaves the -VAT and price data points. Unfortunately I found the OCR to be too unreliable to -extract that text from the documents. Therefore, I had to manually set those two -data points for each ticket. - -Neat, we now have all parking tickets organized on Paperless-ngx! In order to -create a report, we can just query the Paperless-ngx API! - -# Using the Paperless-ngx API - -The remaining part of this post will be code explanations. You can find the full -open-sourced Python code -[here](https://git.kun.is/pim/parking-expenses/src/commit/645fd8b4e4e46bc8f3d8ba831ca28261cd3edb39/main.py). - -Paperless-ngx has [an API](https://docs.paperless-ngx.com/api/) we can now use -to query our parking tickets. I found the filtering logic pretty hard to -understand. Therefore I simply looked at what API calls the web UI makes with -the query I needed. This turned out to be something like this: - -```python -response = requests.get( - f"{paperless_ngx_url}/api/documents/?page_size=50&query=created:[{start_date} TO {end_date}]&tags__id__all={FILTER_TAG_ID}&correspondent__id__in={FILTER_CORRESPONDENT_ID}", - headers={"Authorization": f"Token {token}"}, -) -``` - -In the above code snippet, `FILTER_TAG_ID` is the ID of the "Parking ticket" -label and `FILTER_CORRESPONDENT_ID` is the ID of the correspondent (I included -this to filter out potential unrelated parking tickets). - -Perfect, this returns a JSON list of parking tickets we can use to create the -report! - -# Creating the Report - -All that's left is to fill in an Excel template now. This actually proved to be -the most annoying part but also the most boring part. I used the -[OpenPyXL](https://openpyxl.readthedocs.io/en/stable/) Python library to -manipulate the Excel sheets and basically just add a row for each parking -ticket. For code, check -[here](https://git.kun.is/pim/parking-expenses/src/commit/645fd8b4e4e46bc8f3d8ba831ca28261cd3edb39/main.py#L104). - -To convert the Excel sheet to a PDF, I used the (apparently now deprecated) -[unoconv](https://github.com/unoconv/unoconv) utility. It uses LibreOffice under -the hood for the conversion, which adds a 2GiB dependency for the project 🫠. - -Finally, to merge the report PDF with photos of the parking tickets, I used the -[PyPDF2](https://pypdf2.readthedocs.io/en/3.x/) Python library. - -I would now show some example output, but it contains quite some sensitive -information so unfortunately I can't... But I never had any complaints from the -finance department, so it worked great! diff --git a/src/_posts/parking-expenses-automation/ticket.png b/src/_posts/parking-expenses-automation/ticket.png deleted file mode 100644 index 2665ac3..0000000 Binary files a/src/_posts/parking-expenses-automation/ticket.png and /dev/null differ diff --git a/src/now.md b/src/now.md index 34e4efd..f0e58dd 100644 --- a/src/now.md +++ b/src/now.md @@ -6,12 +6,7 @@ excerpt: Things I am working on now comments: false --- -_Last updated: 2025-06-07_ +*Last updated: 2024-05-11* -- I'm **hacking** on my [k3s](https://k3s.io/) cluster using - [❄️ NixOS ❄️](https://nixos.org/) and [Kubenix](https://kubenix.org/) -- I'm **playing** Hades - -What games did I finish in 2025? - -- Celeste +- I'm **hacking** on my [k3s](https://k3s.io/) cluster using [❄️ NixOS ❄️](https://nixos.org/) and [Kubenix](https://kubenix.org/) +- I'm **playing** [🧩 The Talos Principle 2 🤖](https://store.steampowered.com/app/835960/The_Talos_Principle_2/)