WIP infra 2024 snapshot

src/_posts/infra-snapshot-2024/2025-01-12-infra-snapshot-2024.md

@@ -0,0 +1,245 @@
+---
+layout: post
+title: Home Lab Infrastructure Snapshot 2024
+date: 2025-01-12 20:49:00 Europe/Amsterdam
+categories: infrastructure homelab
+---
+
+If you are reading this when this post was just published: fear not, you did not
+just travel back in time to the glorious old days of '24. No, I was simply too
+~~lazy~~ busy to write this post last year. Busy with what? Well, I basically
+changed every part of my infrastructure the past year obviously. This is all due
+to the fact I got infected by the NixOS-virus!
+
+This post is a follow-up of a similar post from 2023 where I described my whole
+home lab as it was back then. In this post I will mainly focus on the things
+that changed compared to then.
+
+# Hardware and Operating Systems
+
+## Hardware
+
+It seems I was rather frugal this year, as I did not buy any new servers or
+replaced existing ones.
+
+However, as we will see in a bit, I am now running Linux on my gaming PC and
+deploying it just like I would do with any one of my servers. Therefore, I will
+take a moment to gloss over the hardware of that machine:
+
+- **Motherboard**: Gigabyte B550 GAMING X V2
+- **CPU**: AMD Ryzen 5 5600
+- **GPU**: AMD Radeon RX 6700 XT
+
+Here is a picture of its guts:
+
+![The inside of my gaming PC showing its components.](gamepc.jpeg)
+
+You probably noticed the random components inside it, which are fixed just
+securely enough to not cause a fire. These are actually parts of a
+[DIY PiKVM V2](https://docs.pikvm.org/v2/). PiKVM is a cool project to transform
+your Raspberry Pi into a KVM (Keyboard, Video and Mouse). A KVM, in its most
+basic form, allows you to see the video output and control the mouse and
+keyboard of a system remotely.
+
+In the bottom right part of the picture, connected to the ribbon cable, you can
+see the part that allows the Raspberry Pi to power on/off the PC, reset the PC,
+read power status and read disk activity, all via its GPIO! In the top right
+part of the picture you can see the component that converts the PC's HDMI output
+to CSI, which the Pi supports. Just behind that component, but out of view, the
+Pi is secured.
+
+PiKVM has some more neat features: it can emulate a USB storage device which I
+can boot off of. This allows me, for example, to run a memory test or tweak BIOS
+settings.
+
+One other piece of hardware is worth mentioning. Similar to the PiKVM, I also
+bought a
+[Sipeed NanoKVM](https://wiki.sipeed.com/hardware/en/kvm/NanoKVM/introduction.html).
+I opted for the Lite variant, which is only able to read HDMI and interface with
+the hosts USB. It can't control the power like the PiKVM that I built. For now I
+attached it to one of my Kubernetes nodes.
+
+![Unboxing of the NanoKVM lite.](nanokvm.png) _Unboxing of the NanoKVM Lite,
+[courtesy of Sipeed](https://github.com/sipeed/sipeed_wiki/blob/main/docs/hardware/assets/NanoKVM/unbox/lite_ubox.png)._
+
+## Operating Systems (or, the NixOS propaganda)
+
+For people that know me in real life, this section will not be a surprise. In
+late 2023 I started getting interested in NixOS and in 2024 I worked hard to
+install it on any piece of hardware I could get my hands on.
+
+NixOS enables me to apply my Infrastrucutre-as-Code spirit to the OS and take it
+to the extreme. I used to customize an existing OS (mainly Debian) with Ansible
+to configure it to my liking. This works to some extent: the steps needed to get
+to my desired configuration are codified! However, after using Ansible for some
+time I started seeing drawbacks.
+
+Ansible likes to advertise itself as declarative (because it's YAML, which is
+declarative by nature right?). Sure, all steps _should_ be declarative and
+idempotent (they can be run repeatedly without changes). But when you do
+anything non-trivial, you start to introduce dependencies between different
+tasks. Suddenly, your tasks cannot be run just on their own anymore! Together, a
+certain Ansible role should indeed still be idempotent, but I often have to
+think about the current state of my machines still.
+
+In contrast, NixOS is configured entirely declaratively. You don't have to tell
+NixOS how to go from configuration A to configuration B, this is something NixOS
+will figure out for you. Not convinced? This is how you get a fully functional
+Kubernetes node on NixOS:
+
+```nix
+{
+    config.services.k3s.enable = true;
+}
+```
+
+### Colmena
+
+Deploying changes on a single NixOS host works fine with the `nixos-rebuild`
+tool. However, if you manage multiple NixOS servers, a bit more comfort is
+quickly desired. That's why I am happily using the
+[Colmena](https://github.com/zhaofengli/colmena) deployment tool. Below is an
+example deployment for my servers.
+
+```shell
+❯ colmena apply  --experimental-flake-eval --on @server
+[INFO ] Using flake: git+file:///home/pim/git/nixos-configs
+[WARN ] Using direct flake evaluation (experimental)
+[INFO ] Enumerating nodes...
+[INFO ] Selected 4 out of 6 hosts.
+        ✅ 2m All done!
+  (...) ✅ 37s Evaluated warwick, atlas, lewis, and jefke
+  atlas ✅ 0s Built "/nix/store/r0jpg2nrqdnk9gywxvzrh2n3lrwhzy56-nixos-system-atlas-24.11pre-git"
+warwick ✅ 4s Built "/nix/store/kfbm5c2fqd2xv6lasvb2nhc8g815hl79-nixos-system-warwick-24.11pre-git" on target node
+  lewis ✅ 0s Built "/nix/store/h238ly237srjil0fdxzrj29ib6blcmlg-nixos-system-lewis-24.11pre-git"
+  jefke ✅ 0s Built "/nix/store/b7pnan3wmgk3y0193rka95i82sl33xpc-nixos-system-jefke-24.11pre-git"
+  atlas ✅ 10s Pushed system closure
+  jefke ✅ 5s Pushed system closure
+  lewis ✅ 9s Pushed system closure
+warwick ✅ 53s Uploaded keys (pre-activation)
+  jefke ✅ 49s Uploaded keys (pre-activation)
+  lewis ✅ 47s Uploaded keys (pre-activation)
+  atlas ✅ 46s Uploaded keys (pre-activation)
+  jefke ✅ 9s Activation successful
+  atlas ✅ 18s Activation successful
+warwick ✅ 4s Activation successful
+  lewis ✅ 16s Activation successful
+```
+
+Notice the use of `@server`, which only deploys hosts thare are tagged with
+`server`. I also deploy my laptop and gaming PC with this tool, so this is very
+handy. Also notice that the host `warwick` is built on the target node. This is
+because `warwick` is a Raspberry Pi 4 with an ARM architecture. The system is
+therefore compiled on the remote.
+
+## Virtualization
+
+I had two main purposes for virtual machines: security and host network
+isolation. I used to run Docker Swarm on my servers, which messes around with
+your `iptables` rules. Virtual machines were therefore a way separate that from
+the main host.
+
+However, I found that VMs are pretty difficult to manage. It's difficult to
+define them in an infrastructure-as-code way. To this end, I used Terraform with
+the pretty buggy
+[terraform-provider-libvirt](https://github.com/dmacvicar/terraform-provider-libvirt)
+provider, but this was not a smooth experience. I also found them running out of
+memory quite often. And while I could have invested time into something like
+memory ballooning, I didn't really want to spend my time with that.
+
+So now my servers are completely VM-free!
+
+## Container Orchestration
+
+Alongside the operating system, my container clustering setup changed the most
+this year.
+
+Before this year, I was using
+[Docker Swarm](https://docs.docker.com/engine/swarm/) for container clustering.
+The main benefit for me, was its similiarities with Docker Compose, which I was
+using at the time. Unfortunately, Docker Swarm is not widely used and doesn't
+seem well maintained either. Also, a feature I was really missing was the option
+for "distributed" storage that syncs data between nodes to mitigate hardware
+failure.
+
+With Docker Swarm out of the option, I needed to choose another solution.
+Initially, I wanted to use [Hashicorp Nomad](https://www.nomadproject.io/).
+Unfortunately, Nomad is no longer open source software, so this is out of the
+question for me. Then, apart from some smaller projects, I really only had one
+option left: **Kubernetes**!
+
+Below I outline some of the components I use in my Kubernetes setup that make it
+tick.
+
+### k3s
+
+I opted for the [k3s](https://k3s.io/) Kubernetes "distribution", because I
+wanted to start simple and as you saw in
+[a previous section](#operating-systems-or-the-nixos-propaganda), it's super
+simple to enable on NixOS. k3s has all the Kubernetes components out-of-the-box
+to run a single-node cluster.
+
+In the future, I might swap to using NixOS' `services.kubernetes`
+[options](https://search.nixos.org/options?channel=24.11&from=0&size=50&sort=relevance&type=packages&query=services.kubernetes)
+as I hear these work great as well and gives you more control over what runs.
+
+### MetalLB
+
+If you use the Kubernetes platform of a cloud provider, like
+[Microsoft's AKS](https://learn.microsoft.com/en-us/azure/aks/),
+[Google's GKE](https://cloud.google.com/kubernetes-engine) or
+[Amazon's EKS](https://aws.amazon.com/eks/) (all really inspiring names,
+probably great to convince your upper management), they provide network load
+balancers by default. These are great, because they simplify exposing services
+outside of the cluster. Unfortunately, bare-metal Kubernetes lacks a load
+balancer and this is where [MetalLB](https://metallb.io/) comes in.
+
+MetalLB works by assigning locally-routable IP addresses to Kubernetes services.
+It has two methods to do this: either via ARP or via BGP. I opted for ARP to not
+overcomplicate my network setup. To use ARP with MetalLB, you simply have to
+reserve some IP address space for MetalLB to play with. It will dynamically
+assign these IPs to Kubernetes services and advertise these via ARP.
+
+### Longhorn
+
+Similar to MetalLB, [Longhorn](https://longhorn.io/) fills a gap that Kubernetes
+has on bare-metal deployments. Without Longhorn, any Kubernetes storage volume
+is tied to a particular host. This is very problematic when you want to move
+containers around, or if a physical server dies.
+
+Longhorn fixes this by replicating block storage across multiple Kubernetes
+nodes. Want to take down a node? No problem, the workloads can move to another
+node with the same data. Longhorn can also create periodic backups, which I
+personally backup off-site to [BorgBase](https://www.borgbase.com/).
+
+### Cert-manager
+
+Another amazing part of my Kubernetes setup is
+[Cert-manager](https://cert-manager.io/). Cert-manager automatically manages TLS
+certificates that are needed for your Kubernetes deployments.
+
+Usage of Cert-manager is super simple. First you have to set up a certificate
+issuer, like Let's Encrypt. Then you can request Cert-manager to automatically
+provision certificates. I use Kubernetes ingresses, and can simply give it the
+`cert-manager.io/cluster-issuer` annotation. Then, Cert-manager will use the
+`host` field to request the certificate.
+
+I abstracted this a bit, and this is for example the ingress definition for the
+blog you are reading right now:
+
+```nix
+{
+  lab.ingresses.blog = {
+    host = "pim.kun.is";
+
+    service = {
+      name = "blog";
+      portName = "web";
+    };
+  };
+}
+```
+
+### Tailscale
+
+## Workloads

src/_posts/parking-expenses-automation/2025-03-02-parking-expenses-automation.md

@@ -1,102 +0,0 @@
----
-layout: post
-title: Automating a Parking Expenses Report
-date: 2025-03-02 14:38:00 Europe/Amsterdam
-categories: paperless paperless-ngx nix python python3
----
-
-For the past year, I was consulting at a place where I could request
-reimbursement of parking expenses. This had to be claimed using a standard Excel
-template.
-
-Doing this manually every month seemed boring, so I automated a significant part
-of that:
-
-- Fetch parking costs for a certain month from my Paperless-ngx instance
-- Calculate costs after taxes and a grand total
-- Fill in an Excel sheet with the details
-- Convert the Excel sheet to PDF and append photos of parking costs
-
-Note: I don't expect anybody to have the same requirements as me here, but
-hopefully pieces can be useful to some. The open-sourced project can be found
-[here](https://git.kun.is/pim/parking-expenses).
-
-# Organizing the Parking Costs
-
-Each time I would park, I would accumulate a parking ticket. This ticket
-contains the following information:
-
-- Date and time
-- Car license plate
-- Price
-- VAT
-
-In order to automate the report, I had to have this information in a structured
-way. Therefore, I turned to [Paperless-ngx](https://docs.paperless-ngx.com/).
-Paperless-ngx is an self-hostable open-source service that helps digitalizing
-paper documents. Third-party mobile apps help quickly scanning documents and
-uploading them to the server. This is exactly what I did with the parking
-tickets as well: each time I received a ticket, I would scan it with the
-Paperless app.
-
-Paperless-ngx uses Optical Character Recognition (OCR) to cleverly extract the
-date of the ticket automatically. I also created a custom label to indicate a
-parking ticket, and I added custom fields to indicate a ticket's cost and VAT
-amount. You can see this information in the screenshot below.
-
-![foo](ticket.png) _Translated from Dutch: aanmaakdatum means creation date,
-parkeerkaart means parking ticket, BTW means VAT and bedrag means cost_
-
-Paperless-ngx will also automatically learn to label the tickets with the
-"parking ticket" label, based on the document's content. That just leaves the
-VAT and price data points. Unfortunately I found the OCR to be too unreliable to
-extract that text from the documents. Therefore, I had to manually set those two
-data points for each ticket.
-
-Neat, we now have all parking tickets organized on Paperless-ngx! In order to
-create a report, we can just query the Paperless-ngx API!
-
-# Using the Paperless-ngx API
-
-The remaining part of this post will be code explanations. You can find the full
-open-sourced Python code
-[here](https://git.kun.is/pim/parking-expenses/src/commit/645fd8b4e4e46bc8f3d8ba831ca28261cd3edb39/main.py).
-
-Paperless-ngx has [an API](https://docs.paperless-ngx.com/api/) we can now use
-to query our parking tickets. I found the filtering logic pretty hard to
-understand. Therefore I simply looked at what API calls the web UI makes with
-the query I needed. This turned out to be something like this:
-
-```python
-response = requests.get(
-    f"{paperless_ngx_url}/api/documents/?page_size=50&query=created:[{start_date} TO {end_date}]&tags__id__all={FILTER_TAG_ID}&correspondent__id__in={FILTER_CORRESPONDENT_ID}",
-    headers={"Authorization": f"Token {token}"},
-)
-```
-
-In the above code snippet, `FILTER_TAG_ID` is the ID of the "Parking ticket"
-label and `FILTER_CORRESPONDENT_ID` is the ID of the correspondent (I included
-this to filter out potential unrelated parking tickets).
-
-Perfect, this returns a JSON list of parking tickets we can use to create the
-report!
-
-# Creating the Report
-
-All that's left is to fill in an Excel template now. This actually proved to be
-the most annoying part but also the most boring part. I used the
-[OpenPyXL](https://openpyxl.readthedocs.io/en/stable/) Python library to
-manipulate the Excel sheets and basically just add a row for each parking
-ticket. For code, check
-[here](https://git.kun.is/pim/parking-expenses/src/commit/645fd8b4e4e46bc8f3d8ba831ca28261cd3edb39/main.py#L104).
-
-To convert the Excel sheet to a PDF, I used the (apparently now deprecated)
-[unoconv](https://github.com/unoconv/unoconv) utility. It uses LibreOffice under
-the hood for the conversion, which adds a 2GiB dependency for the project 🫠.
-
-Finally, to merge the report PDF with photos of the parking tickets, I used the
-[PyPDF2](https://pypdf2.readthedocs.io/en/3.x/) Python library.
-
-I would now show some example output, but it contains quite some sensitive
-information so unfortunately I can't... But I never had any complaints from the
-finance department, so it worked great!

src/now.md

@@ -6,9 +6,7 @@ excerpt: Things I am working on now
 comments: false
 ---
 
-_Last updated: 2025-03-02_
+*Last updated: 2024-05-11*
 
-- I'm **hacking** on my [k3s](https://k3s.io/) cluster using
+- I'm **hacking** on my [k3s](https://k3s.io/) cluster using [❄️ NixOS ❄️](https://nixos.org/) and [Kubenix](https://kubenix.org/)
-  [❄️ NixOS ❄️](https://nixos.org/) and [Kubenix](https://kubenix.org/)
+- I'm **playing** [🧩 The Talos Principle 2 🤖](https://store.steampowered.com/app/835960/The_Talos_Principle_2/)
-- I'm **playing**
-  [🧩 The Binding of Isaac: Repentance 🔪](https://www.nicalis.com/games/thebindingofisaacrepentance)