diff --git a/jekyll/_posts/infrastructure-snapshot/2023-08-13-infrastructure-snapshot.md b/jekyll/_posts/infrastructure-snapshot/2023-08-13-infrastructure-snapshot.md new file mode 100644 index 0000000..f9abcb2 --- /dev/null +++ b/jekyll/_posts/infrastructure-snapshot/2023-08-13-infrastructure-snapshot.md @@ -0,0 +1,283 @@ +--- +layout: post +title: Home Lab Infrastructure Snapshot August 2023 +date: 2023-08-27 22:23:00 Europe/Amsterdam +categories: infrastructure homelab +--- + +I have been meaning to write about the current state of my home lab infrastructure for a while now. +Now that the most important parts are quite stable, I think the opportunity is ripe. +I expect this post to get quite long, so I might have to leave out some details along the way. + +This post will be a starting point for future infrastructure snapshots which I can hopefully put out periodically. +That is, if there is enough worth talking about. + +Keep an eye out for the icon, which links to the source code and configuration of anything mentioned. +Oh yeah, did I mention everything I do is open source? + +# Networking and Infrastructure Overview + +## Hardware and Operating Systems + +Let's start with the basics: what kind of hardware do I use for my home lab? +The most important servers are my three [Gigabyte Brix GB-BLCE-4105](https://www.gigabyte.com/Mini-PcBarebone/GB-BLCE-4105-rev-10). +Two of them have 16 GB of memory, and one 8 GB. +I named these servers as follows: +- **Atlas**: because this server was going to "lift" a lot of virtual machines. +- **Lewis**: we started out with a "Max" server named after the Formula 1 driver Max Verstappen, but it kind of became an unmanagable behemoth without infrastructure-as-code. Our second server we subsequently named Lewis after his colleague Lewis Hamilton. Note: people around me vetoed these names and I am no F1 fan! +- **Jefke**: it's a funny Belgian name. That's all. + +Here is a picture of them sitting in their cosy closet: + +![A picture of my servers.](servers.jpeg) + +If you look look to the left, you will also see a Raspberry pi 4B. +I use this Pi to do some rudimentary monitoring whether servers and services are running. +More on this in the relevant section below. +The Pi is called **Iris** because it's a messenger for the other servers. + +I used to run Ubuntu on these systems, but I have since migrated away to Debian. +The main reasons were Canonical [putting advertisements in my terminal](https://askubuntu.com/questions/1434512/how-to-get-rid-of-ubuntu-pro-advertisement-when-updating-apt) and pushing Snap which has a [proprietry backend](https://hackaday.com/2020/06/24/whats-the-deal-with-snap-packages/). +Two of my servers run the newly released Debian Bookworm, while one still runs Debian Bullseye. + +## Networking + +For networking, I wanted hypervisors and virtual machines separated by VLANs for security reasons. +The following picture shows a simplified view of the VLANs present in my home lab: + +![Picture showing the VLANS in my home lab.](vlans.png) + +All virtual machines are connected to a virtual bridge which tags network traffic with the DMZ VLAN. +The hypervisors VLAN is used for traffic to and from the hypervisors. +Devices from the hypervisors VLAN are allowed to connect to devices in the DMZ, but not vice versa. +The hypervisors are connected to a switch using a trunk link, allows both DMZ and hypervisors traffic. + +I realised the above design using ifupdown. +Below is the configuration for each hypervisor, which creates a new `enp3s0.30` interface with all DMZ traffic from the `enp3s0` interface [](https://git.kun.is/home/hypervisors/src/commit/71b96d462116e4160b6467533fc476f3deb9c306/ansible/dmz.conf.j2). + +```text +auto enp3s0.30 +iface enp3s0.30 inet manual +iface enp3s0.30 inet6 auto + accept_ra 0 + dhcp 0 + request_prefix 0 + privext 0 + pre-up sysctl -w net/ipv6/conf/enp3s0.30/disable_ipv6=1 +``` + +This configuration seems more complex than it actually is. +Most of it is to make sure the interface is not assigned an IPv4/6 address on the hypervisor host. +The magic `.30` at the end of the interface name makes this interface tagged with VLAN ID 30 (DMZ for me). + +Now that we have an interface tagged for the DMZ VLAN, we can create a bridge where future virtual machines can connect to: + +```text +auto dmzbr +iface dmzbr inet manual + bridge_ports enp3s0.30 + bridge_stp off +iface dmzbr inet6 auto + accept_ra 0 + dhcp 0 + request_prefix 0 + privext 0 + pre-up sysctl -w net/ipv6/conf/dmzbr/disable_ipv6=1 +``` + +Just like the previous config, this is quite bloated because I don't want the interface to be assigned an IP address on the host. +Most importantly, the `bridge_ports enp3s0.30` line here makes this interface a virtual bridge for the `enp3s0.30` interface. + +And voilà, we now have a virtual bridge on each machine, where only DMZ traffic will flow. +Here I verify whether this configuration works: +
+ Show + + +We can see that the two virtual interfaces are created, and are only assigned a MAC address and not a IP address: +```text +root@atlas:~# ip a show enp3s0.30 +4: enp3s0.30@enp3s0: mtu 1500 qdisc noqueue master dmzbr state UP group default qlen 1000 + link/ether d8:5e:d3:4c:70:38 brd ff:ff:ff:ff:ff:ff +5: dmzbr: mtu 1500 qdisc noqueue state UP group default qlen 1000 + link/ether 4e:f7:1f:0f:ad:17 brd ff:ff:ff:ff:ff:ff +``` + +Pinging a VM from a hypervisor works: +```text +root@atlas:~# ping -c1 maestro.dmz +PING maestro.dmz (192.168.30.8) 56(84) bytes of data. +64 bytes from 192.168.30.8 (192.168.30.8): icmp_seq=1 ttl=63 time=0.457 ms +``` + +Pinging a hypervisor from a VM does not work: +```text +root@maestro:~# ping -c1 atlas.hyp +PING atlas.hyp (192.168.40.2) 56(84) bytes of data. + +--- atlas.hyp ping statistics --- +1 packets transmitted, 0 received, 100% packet loss, time 0ms +``` +
+ +## DNS and DHCP + +Now that we have a working DMZ network, let's build on it to get DNS and DHCP working. +This will enable new virtual machines to obtain a static or dynamic IP address and register their host in DNS. +This has actually been incredibly annoying due to our friend [Network address translation (NAT)](https://en.wikipedia.org/wiki/Network_address_translation?useskin=vector). +
+ NAT recap + +Network address translation (NAT) is a function of a router which allows multiple hosts to share a single IP address. +This is needed for IPv4, because IPv4 addresses are scarce and usually one household is only assigned a single IPv4 address. +This is one of the problems IPv6 attempts to solve (mainly by having so many IP addresses that they should never run out). +To solve the problem for IPv4, each host in a network is assigned a private IPv4 address, which can be reused for every network. + +Then, the router must perform address translation. +It does this by keeping track of ports opened by hosts in its private network. +If a packet from the internet arrives at the router for such a port, it forwards this packet to the correct host. +
+ +I would like to host my own DNS on a virtual machine (called **hermes**, more on VMs later) in the DMZ network. +This basically gives two problems: + +1. The upstream DNS server will refer to the public internet-accessible IP address of our DNS server. +This IP-address has no meaning inside the private network due to NAT and the router will reject the packet. +2. Our DNS resolves hosts to their public internet-accessible IP address. +This is similar to the previous problem as the public IP address has no meaning. + +The first problem can be remediated by overriding the location of the DNS server for hosts inside the DMZ network. +This can be achieved on my router, which uses Unbound as its recursive DNS server: + +![Unbound overides for kun.is and dmz domains.](unbound_overrides.png) + +Any DNS requests to Unbound to domains in either `dmz` or `kun.is` will now be forwarded `192.168.30.7` (port 5353). +This is the virtual machine hosting my DNS. + +The second problem can be solved at the DNS server. +We need to do some magic overriding, which [dnsmasq](https://dnsmasq.org/docs/dnsmasq-man.html) is perfect for [](https://git.kun.is/home/hermes/src/commit/488024a7725f2325b8992e7a386b4630023f1b52/ansible/roles/dnsmasq/files/dnsmasq.conf): + +```conf +alias=84.245.14.149,192.168.30.8 +server=/kun.is/192.168.30.7 +``` + +This always overrides the public IPv4 address to the private one. +It also overrides the DNS server for `kun.is` to `192.168.30.7`. + +Finally, behind the dnsmasq server, I run [Powerdns](https://www.powerdns.com/) as authoritative DNS server [](https://git.kun.is/home/hermes/src/branch/master/ansible/roles/powerdns). +I like this DNS server because I can manage it with Terraform [](https://git.kun.is/home/hermes/src/commit/488024a7725f2325b8992e7a386b4630023f1b52/terraform/dns/kun_is.tf). + +Here is a small diagram showing my setup (my networking teacher would probably kill me for this): +![Shitty diagram showing my DNS setup.](nat.png) + +# Virtualization +https://github.com/containrrr/shepherd +Now that we have laid out the basic networking, let's talk virtualization. +Each of my servers are configured to run KVM virtual machines, orchestrated using Libvirt. +Configuration of the physical hypervisor servers, including KVM/Libvirt is done using Ansible. +The VMs are spun up using Terraform and the [dmacvicar/libvirt](https://registry.terraform.io/providers/dmacvicar/libvirt/latest/docs) Terraform provider. + +This all isn't too exciting, except that I created a Terraform module that abstracts the Terraform Libvirt provider for my specific scenario [](https://git.kun.is/home/tf-modules/src/commit/e77d62f4a2a0c3847ffef4434c50a0f40f1fa794/debian/main.tf): +```terraform +module "maestro" { + source = "git::https://git.kun.is/home/tf-modules.git//debian" + name = "maestro" + domain_name = "tf-maestro" + memory = 10240 + mac = "CA:FE:C0:FF:EE:08" +} +``` + +This automatically creates a Debian virtual machines with the properties specified. +It also sets up certificate-based SSH authentication which I talked about [before]({% post_url homebrew-ssh-ca/2023-05-23-homebrew-ssh-ca %}). + +# Clustering + +With virtualization explained, let's move up one level further. +Each of my three physical servers hosts a virtual machine running Docker, which together form a Docker Swarm. +I use Traefik as a reverse proxy which routes requests to the correct container. + +All data is hosted on a single machine and made available to containers using NFS. +This might not be very secure (as NFS is not encrypted and no proper authentication), it is quite fast. + +As of today, I host the following services on my Docker Swarm [](https://git.kun.is/home/shoarma): +- [Forgejo](https://forgejo.org/) as Git server +- [FreshRSS](https://www.freshrss.org/) as RSS aggregator +- [Hedgedoc](https://hedgedoc.org/) as Markdown note-taking +- [Inbucket](https://hedgedoc.org/) for disposable email +- [Cyberchef](https://cyberchef.org/) for the lulz +- [Kitchenowl](https://kitchenowl.org/) for grocery lists +- [Mastodon](https://joinmastodon.org/) for microblogging +- A monitoring stack (read more below) +- [Nextcloud](https://nextcloud.com/) for cloud storage +- [Pihole](https://pi-hole.net/) to block advertisements +- [Radicale](https://radicale.org/v3.html) for calendar and contacts sync +- [Seafile](https://www.seafile.com/en/home/) for cloud storage and sync +- [Shephard](https://github.com/containrrr/shepherd) for automatic container updates +- [Nginx](https://nginx.org/en/) hosting static content (like this page!) +- [Docker Swarm dashboard](https://hub.docker.com/r/charypar/swarm-dashboard/#!) +- [Syncthing](https://syncthing.net/) for file sync + +# CI / CD + +For CI / CD, I run [Concourse CI](https://concourse-ci.org/) in a separate VM. +This is needed, because Concourse heavily uses containers to create reproducible builds. + +Although I should probably use it for more, I currently use my Concourse for three pipelines: + +- A pipeline to build this static website and create a container image of it. +The image is then uploaded to the image registry of my Forgejo instance. +I love it when I can use stuff I previously built :) +The pipeline finally deploys this new image to the Docker Swarm [](https://git.kun.is/pim/static/src/commit/eee4f0c70af6f2a49fabb730df761baa6475db22/pipeline.yml). +- A pipeline to create a Concourse resource that sends Apprise alerts (Concourse-ception?) [](https://git.kun.is/pim/concourse-apprise-notifier/src/commit/b5d4413c1cd432bc856c45ec497a358aca1b8b21/pipeline.yml) +- A pipeline to build a custom Fluentd image with plugins installed [](https://git.kun.is/pim/fluentd) + +# Backups + +To create backups, I use [Borg](https://www.borgbackup.org/). +As I keep all data on one machine, this backup process is quite simple. +In fact, all this data is stored in a single Libvirt volume. +To configure Borg with a simple declarative script, I use [Borgmatic](https://torsion.org/borgmatic/). + +In order to back up the data inside the Libvirt volume, I create a snapshot to a file. +Then I can mount this snapshot in my file system. +The files can then be backed up while the system is still running. +It is also possible to simply back up the Libvirt image, but this takes more time and storage [](https://git.kun.is/home/hypervisors/src/commit/71b96d462116e4160b6467533fc476f3deb9c306/ansible/roles/borg/backup.yml.j2). + +# Monitoring and Alerting + +The last topic I would like to talk about is monitoring and alerting. +This is something I'm still actively improving and only just set up properly. + +## Alerting + +For alerting, I wanted something that runs entirely on my own infrastructure. +I settled for Apprise + Ntfy. + +[Apprise](https://github.com/caronc/apprise) is a server that is able to send notifications to dozens of services. +For application developers, it is thus only necessary to implement the Apprise API to gain access to all these services. +The Apprise API itself is also very simple. +By using Apprise, I can also easily switch to another notification service later. +[Ntfy](https://ntfy.sh/) is free software made for mobile push notifications. + +I use this alerting system in quite a lot of places in my infrastructure, for example when creating backups. + +## Uptime Monitoring + +The first monitoring setup I created, was using [Uptime Kuma](https://github.com/louislam/uptime-kuma). +Uptime Kuma periodically pings a service to see whether it is still running. +You can do a literal ping, test HTTP response codes, check database connectivity and much more. +I use it to check whether my services and VMs are online. +And the best part is, Uptime Kuma supports Apprise so I get push notifications on my phone whenever something goes down! + +## Metrics and Log Monitoring + +A new monitoring system I am still in the process of deploying is focused on metrics and logs. +I plan on creating a separate blog post about this, so keep an eye out on that (for example using RSS :)). +Safe to say, it is no basic ELK stack! + +# Conclusion + +That's it for now! +Hopefully I inspired someone to build something... or how not to :) diff --git a/jekyll/_posts/infrastructure-snapshot/nat.png b/jekyll/_posts/infrastructure-snapshot/nat.png new file mode 100644 index 0000000..0d5f72c Binary files /dev/null and b/jekyll/_posts/infrastructure-snapshot/nat.png differ diff --git a/jekyll/_posts/infrastructure-snapshot/servers.jpeg b/jekyll/_posts/infrastructure-snapshot/servers.jpeg new file mode 100644 index 0000000..b269484 Binary files /dev/null and b/jekyll/_posts/infrastructure-snapshot/servers.jpeg differ diff --git a/jekyll/_posts/infrastructure-snapshot/unbound_overrides.png b/jekyll/_posts/infrastructure-snapshot/unbound_overrides.png new file mode 100644 index 0000000..f94394f Binary files /dev/null and b/jekyll/_posts/infrastructure-snapshot/unbound_overrides.png differ diff --git a/jekyll/_posts/infrastructure-snapshot/vlans.png b/jekyll/_posts/infrastructure-snapshot/vlans.png new file mode 100644 index 0000000..7bf4add Binary files /dev/null and b/jekyll/_posts/infrastructure-snapshot/vlans.png differ