Improve about page
Remove cache and compiled site
This commit is contained in:
parent
77d432ad8b
commit
03cf79e8f0
82 changed files with 321 additions and 1405 deletions
|
|
@ -0,0 +1,281 @@
|
|||
I"ýG<p>I have been meaning to write about the current state of my home lab infrastructure for a while now.
|
||||
Now that the most important parts are quite stable, I think the opportunity is ripe.
|
||||
I expect this post to get quite long, so I might have to leave out some details along the way.</p>
|
||||
|
||||
<p>This post will be a starting point for future infrastructure snapshots which I can hopefully put out periodically.
|
||||
That is, if there is enough worth talking about.</p>
|
||||
|
||||
<p>Keep an eye out for the <i class="fa-solid fa-code-branch"></i> icon, which links to the source code and configuration of anything mentioned.
|
||||
Oh yeah, did I mention everything I do is open source?</p>
|
||||
|
||||
<h1 id="networking-and-infrastructure-overview">Networking and Infrastructure Overview</h1>
|
||||
|
||||
<h2 id="hardware-and-operating-systems">Hardware and Operating Systems</h2>
|
||||
|
||||
<p>Let’s start with the basics: what kind of hardware do I use for my home lab?
|
||||
The most important servers are my three <a href="https://www.gigabyte.com/Mini-PcBarebone/GB-BLCE-4105-rev-10">Gigabyte Brix GB-BLCE-4105</a>.
|
||||
Two of them have 16 GB of memory, and one 8 GB.
|
||||
I named these servers as follows:</p>
|
||||
<ul>
|
||||
<li><strong>Atlas</strong>: because this server was going to “liftâ€<C3A2> a lot of virtual machines.</li>
|
||||
<li><strong>Lewis</strong>: we started out with a “Maxâ€<C3A2> server named after the Formula 1 driver Max Verstappen, but it kind of became an unmanagable behemoth without infrastructure-as-code. Our second server we subsequently named Lewis after his colleague Lewis Hamilton. Note: people around me vetoed these names and I am no F1 fan!</li>
|
||||
<li><strong>Jefke</strong>: it’s a funny Belgian name. That’s all.</li>
|
||||
</ul>
|
||||
|
||||
<p>Here is a picture of them sitting in their cosy closet:</p>
|
||||
|
||||
<p><img src="servers.jpeg" alt="A picture of my servers." /></p>
|
||||
|
||||
<p>If you look look to the left, you will also see a Raspberry pi 4B.
|
||||
I use this Pi to do some rudimentary monitoring whether servers and services are running.
|
||||
More on this in the relevant section below.
|
||||
The Pi is called <strong>Iris</strong> because it’s a messenger for the other servers.</p>
|
||||
|
||||
<p>I used to run Ubuntu on these systems, but I have since migrated away to Debian.
|
||||
The main reasons were Canonical <a href="https://askubuntu.com/questions/1434512/how-to-get-rid-of-ubuntu-pro-advertisement-when-updating-apt">putting advertisements in my terminal</a> and pushing Snap which has a <a href="https://hackaday.com/2020/06/24/whats-the-deal-with-snap-packages/">proprietry backend</a>.
|
||||
Two of my servers run the newly released Debian Bookworm, while one still runs Debian Bullseye.</p>
|
||||
|
||||
<h2 id="networking">Networking</h2>
|
||||
|
||||
<p>For networking, I wanted hypervisors and virtual machines separated by VLANs for security reasons.
|
||||
The following picture shows a simplified view of the VLANs present in my home lab:</p>
|
||||
|
||||
<p><img src="vlans.png" alt="Picture showing the VLANS in my home lab." /></p>
|
||||
|
||||
<p>All virtual machines are connected to a virtual bridge which tags network traffic with the DMZ VLAN.
|
||||
The hypervisors VLAN is used for traffic to and from the hypervisors.
|
||||
Devices from the hypervisors VLAN are allowed to connect to devices in the DMZ, but not vice versa.
|
||||
The hypervisors are connected to a switch using a trunk link, allows both DMZ and hypervisors traffic.</p>
|
||||
|
||||
<p>I realised the above design using ifupdown.
|
||||
Below is the configuration for each hypervisor, which creates a new <code class="language-plaintext highlighter-rouge">enp3s0.30</code> interface with all DMZ traffic from the <code class="language-plaintext highlighter-rouge">enp3s0</code> interface <a href="https://git.kun.is/home/hypervisors/src/commit/71b96d462116e4160b6467533fc476f3deb9c306/ansible/dmz.conf.j2"><i class="fa-solid fa-code-branch"></i></a>.</p>
|
||||
|
||||
<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>auto enp3s0.30
|
||||
iface enp3s0.30 inet manual
|
||||
iface enp3s0.30 inet6 auto
|
||||
accept_ra 0
|
||||
dhcp 0
|
||||
request_prefix 0
|
||||
privext 0
|
||||
pre-up sysctl -w net/ipv6/conf/enp3s0.30/disable_ipv6=1
|
||||
</code></pre></div></div>
|
||||
|
||||
<p>This configuration seems more complex than it actually is.
|
||||
Most of it is to make sure the interface is not assigned an IPv4/6 address on the hypervisor host.
|
||||
The magic <code class="language-plaintext highlighter-rouge">.30</code> at the end of the interface name makes this interface tagged with VLAN ID 30 (DMZ for me).</p>
|
||||
|
||||
<p>Now that we have an interface tagged for the DMZ VLAN, we can create a bridge where future virtual machines can connect to:</p>
|
||||
|
||||
<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>auto dmzbr
|
||||
iface dmzbr inet manual
|
||||
bridge_ports enp3s0.30
|
||||
bridge_stp off
|
||||
iface dmzbr inet6 auto
|
||||
accept_ra 0
|
||||
dhcp 0
|
||||
request_prefix 0
|
||||
privext 0
|
||||
pre-up sysctl -w net/ipv6/conf/dmzbr/disable_ipv6=1
|
||||
</code></pre></div></div>
|
||||
|
||||
<p>Just like the previous config, this is quite bloated because I don’t want the interface to be assigned an IP address on the host.
|
||||
Most importantly, the <code class="language-plaintext highlighter-rouge">bridge_ports enp3s0.30</code> line here makes this interface a virtual bridge for the <code class="language-plaintext highlighter-rouge">enp3s0.30</code> interface.</p>
|
||||
|
||||
<p>And voilà , we now have a virtual bridge on each machine, where only DMZ traffic will flow.
|
||||
Here I verify whether this configuration works:</p>
|
||||
<details>
|
||||
<summary>Show</summary>
|
||||
|
||||
|
||||
We can see that the two virtual interfaces are created, and are only assigned a MAC address and not a IP address:
|
||||
```text
|
||||
root@atlas:~# ip a show enp3s0.30
|
||||
4: enp3s0.30@enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master dmzbr state UP group default qlen 1000
|
||||
link/ether d8:5e:d3:4c:70:38 brd ff:ff:ff:ff:ff:ff
|
||||
5: dmzbr: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
|
||||
link/ether 4e:f7:1f:0f:ad:17 brd ff:ff:ff:ff:ff:ff
|
||||
```
|
||||
|
||||
Pinging a VM from a hypervisor works:
|
||||
```text
|
||||
root@atlas:~# ping -c1 maestro.dmz
|
||||
PING maestro.dmz (192.168.30.8) 56(84) bytes of data.
|
||||
64 bytes from 192.168.30.8 (192.168.30.8): icmp_seq=1 ttl=63 time=0.457 ms
|
||||
```
|
||||
|
||||
Pinging a hypervisor from a VM does not work:
|
||||
```text
|
||||
root@maestro:~# ping -c1 atlas.hyp
|
||||
PING atlas.hyp (192.168.40.2) 56(84) bytes of data.
|
||||
|
||||
--- atlas.hyp ping statistics ---
|
||||
1 packets transmitted, 0 received, 100% packet loss, time 0ms
|
||||
```
|
||||
</details>
|
||||
|
||||
<h2 id="dns-and-dhcp">DNS and DHCP</h2>
|
||||
|
||||
<p>Now that we have a working DMZ network, let’s build on it to get DNS and DHCP working.
|
||||
This will enable new virtual machines to obtain a static or dynamic IP address and register their host in DNS.
|
||||
This has actually been incredibly annoying due to our friend <a href="https://en.wikipedia.org/wiki/Network_address_translation?useskin=vector">Network address translation (NAT)</a>.</p>
|
||||
<details>
|
||||
<summary>NAT recap</summary>
|
||||
|
||||
Network address translation (NAT) is a function of a router which allows multiple hosts to share a single IP address.
|
||||
This is needed for IPv4, because IPv4 addresses are scarce and usually one household is only assigned a single IPv4 address.
|
||||
This is one of the problems IPv6 attempts to solve (mainly by having so many IP addresses that they should never run out).
|
||||
To solve the problem for IPv4, each host in a network is assigned a private IPv4 address, which can be reused for every network.
|
||||
|
||||
Then, the router must perform address translation.
|
||||
It does this by keeping track of ports opened by hosts in its private network.
|
||||
If a packet from the internet arrives at the router for such a port, it forwards this packet to the correct host.
|
||||
</details>
|
||||
|
||||
<p>I would like to host my own DNS on a virtual machine (called <strong>hermes</strong>, more on VMs later) in the DMZ network.
|
||||
This basically gives two problems:</p>
|
||||
|
||||
<ol>
|
||||
<li>The upstream DNS server will refer to the public internet-accessible IP address of our DNS server.
|
||||
This IP-address has no meaning inside the private network due to NAT and the router will reject the packet.</li>
|
||||
<li>Our DNS resolves hosts to their public internet-accessible IP address.
|
||||
This is similar to the previous problem as the public IP address has no meaning.</li>
|
||||
</ol>
|
||||
|
||||
<p>The first problem can be remediated by overriding the location of the DNS server for hosts inside the DMZ network.
|
||||
This can be achieved on my router, which uses Unbound as its recursive DNS server:</p>
|
||||
|
||||
<p><img src="unbound_overrides.png" alt="Unbound overides for kun.is and dmz domains." /></p>
|
||||
|
||||
<p>Any DNS requests to Unbound to domains in either <code class="language-plaintext highlighter-rouge">dmz</code> or <code class="language-plaintext highlighter-rouge">kun.is</code> will now be forwarded <code class="language-plaintext highlighter-rouge">192.168.30.7</code> (port 5353).
|
||||
This is the virtual machine hosting my DNS.</p>
|
||||
|
||||
<p>The second problem can be solved at the DNS server.
|
||||
We need to do some magic overriding, which <a href="https://dnsmasq.org/docs/dnsmasq-man.html">dnsmasq</a> is perfect for <a href="https://git.kun.is/home/hermes/src/commit/488024a7725f2325b8992e7a386b4630023f1b52/ansible/roles/dnsmasq/files/dnsmasq.conf"><i class="fa-solid fa-code-branch"></i></a>:</p>
|
||||
|
||||
<div class="language-conf highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">alias</span>=<span class="m">84</span>.<span class="m">245</span>.<span class="m">14</span>.<span class="m">149</span>,<span class="m">192</span>.<span class="m">168</span>.<span class="m">30</span>.<span class="m">8</span>
|
||||
<span class="n">server</span>=/<span class="n">kun</span>.<span class="n">is</span>/<span class="m">192</span>.<span class="m">168</span>.<span class="m">30</span>.<span class="m">7</span>
|
||||
</code></pre></div></div>
|
||||
|
||||
<p>This always overrides the public IPv4 address to the private one.
|
||||
It also overrides the DNS server for <code class="language-plaintext highlighter-rouge">kun.is</code> to <code class="language-plaintext highlighter-rouge">192.168.30.7</code>.</p>
|
||||
|
||||
<p>Finally, behind the dnsmasq server, I run <a href="https://www.powerdns.com/">Powerdns</a> as authoritative DNS server <a href="https://git.kun.is/home/hermes/src/branch/master/ansible/roles/powerdns"><i class="fa-solid fa-code-branch"></i></a>.
|
||||
I like this DNS server because I can manage it with Terraform <a href="https://git.kun.is/home/hermes/src/commit/488024a7725f2325b8992e7a386b4630023f1b52/terraform/dns/kun_is.tf"><i class="fa-solid fa-code-branch"></i></a>.</p>
|
||||
|
||||
<p>Here is a small diagram showing my setup (my networking teacher would probably kill me for this):
|
||||
<img src="nat.png" alt="Shitty diagram showing my DNS setup." /></p>
|
||||
|
||||
<h1 id="virtualization">Virtualization</h1>
|
||||
<p>https://github.com/containrrr/shepherd
|
||||
Now that we have laid out the basic networking, let’s talk virtualization.
|
||||
Each of my servers are configured to run KVM virtual machines, orchestrated using Libvirt.
|
||||
Configuration of the physical hypervisor servers, including KVM/Libvirt is done using Ansible.
|
||||
The VMs are spun up using Terraform and the <a href="https://registry.terraform.io/providers/dmacvicar/libvirt/latest/docs">dmacvicar/libvirt</a> Terraform provider.</p>
|
||||
|
||||
<p>This all isn’t too exciting, except that I created a Terraform module that abstracts the Terraform Libvirt provider for my specific scenario <a href="https://git.kun.is/home/tf-modules/src/commit/e77d62f4a2a0c3847ffef4434c50a0f40f1fa794/debian/main.tf"><i class="fa-solid fa-code-branch"></i></a>:</p>
|
||||
<div class="language-terraform highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">module</span> <span class="s2">"maestro"</span> <span class="p">{</span>
|
||||
<span class="nx">source</span> <span class="p">=</span> <span class="s2">"git::https://git.kun.is/home/tf-modules.git//debian"</span>
|
||||
<span class="nx">name</span> <span class="p">=</span> <span class="s2">"maestro"</span>
|
||||
<span class="nx">domain_name</span> <span class="p">=</span> <span class="s2">"tf-maestro"</span>
|
||||
<span class="nx">memory</span> <span class="p">=</span> <span class="mi">10240</span>
|
||||
<span class="nx">mac</span> <span class="p">=</span> <span class="s2">"CA:FE:C0:FF:EE:08"</span>
|
||||
<span class="p">}</span>
|
||||
</code></pre></div></div>
|
||||
|
||||
<p>This automatically creates a Debian virtual machines with the properties specified.
|
||||
It also sets up certificate-based SSH authentication which I talked about <a href="/src/ssh/terraform/ansible/2023/05/23/homebrew-ssh-ca.html">before</a>.</p>
|
||||
|
||||
<h1 id="clustering">Clustering</h1>
|
||||
|
||||
<p>With virtualization explained, let’s move up one level further.
|
||||
Each of my three physical servers hosts a virtual machine running Docker, which together form a Docker Swarm.
|
||||
I use Traefik as a reverse proxy which routes requests to the correct container.</p>
|
||||
|
||||
<p>All data is hosted on a single machine and made available to containers using NFS.
|
||||
This might not be very secure (as NFS is not encrypted and no proper authentication), it is quite fast.</p>
|
||||
|
||||
<p>As of today, I host the following services on my Docker Swarm <a href="https://git.kun.is/home/shoarma"><i class="fa-solid fa-code-branch"></i></a>:</p>
|
||||
<ul>
|
||||
<li><a href="https://forgejo.org/">Forgejo</a> as Git server</li>
|
||||
<li><a href="https://www.freshrss.org/">FreshRSS</a> as RSS aggregator</li>
|
||||
<li><a href="https://hedgedoc.org/">Hedgedoc</a> as Markdown note-taking</li>
|
||||
<li><a href="https://hedgedoc.org/">Inbucket</a> for disposable email</li>
|
||||
<li><a href="https://cyberchef.org/">Cyberchef</a> for the lulz</li>
|
||||
<li><a href="https://kitchenowl.org/">Kitchenowl</a> for grocery lists</li>
|
||||
<li><a href="https://joinmastodon.org/">Mastodon</a> for microblogging</li>
|
||||
<li>A monitoring stack (read more below)</li>
|
||||
<li><a href="https://nextcloud.com/">Nextcloud</a> for cloud storage</li>
|
||||
<li><a href="https://pi-hole.net/">Pihole</a> to block advertisements</li>
|
||||
<li><a href="https://radicale.org/v3.html">Radicale</a> for calendar and contacts sync</li>
|
||||
<li><a href="https://www.seafile.com/en/home/">Seafile</a> for cloud storage and sync</li>
|
||||
<li><a href="https://github.com/containrrr/shepherd">Shephard</a> for automatic container updates</li>
|
||||
<li><a href="https://nginx.org/en/">Nginx</a> hosting static content (like this page!)</li>
|
||||
<li><a href="https://hub.docker.com/r/charypar/swarm-dashboard/#!">Docker Swarm dashboard</a></li>
|
||||
<li><a href="https://syncthing.net/">Syncthing</a> for file sync</li>
|
||||
</ul>
|
||||
|
||||
<h1 id="ci--cd">CI / CD</h1>
|
||||
|
||||
<p>For CI / CD, I run <a href="https://concourse-ci.org/">Concourse CI</a> in a separate VM.
|
||||
This is needed, because Concourse heavily uses containers to create reproducible builds.</p>
|
||||
|
||||
<p>Although I should probably use it for more, I currently use my Concourse for three pipelines:</p>
|
||||
|
||||
<ul>
|
||||
<li>A pipeline to build this static website and create a container image of it.
|
||||
The image is then uploaded to the image registry of my Forgejo instance.
|
||||
I love it when I can use stuff I previously built :)
|
||||
The pipeline finally deploys this new image to the Docker Swarm <a href="https://git.kun.is/pim/static/src/commit/eee4f0c70af6f2a49fabb730df761baa6475db22/pipeline.yml"><i class="fa-solid fa-code-branch"></i></a>.</li>
|
||||
<li>A pipeline to create a Concourse resource that sends Apprise alerts (Concourse-ception?) <a href="https://git.kun.is/pim/concourse-apprise-notifier/src/commit/b5d4413c1cd432bc856c45ec497a358aca1b8b21/pipeline.yml"><i class="fa-solid fa-code-branch"></i></a></li>
|
||||
<li>A pipeline to build a custom Fluentd image with plugins installed <a href="https://git.kun.is/pim/fluentd"><i class="fa-solid fa-code-branch"></i></a></li>
|
||||
</ul>
|
||||
|
||||
<h1 id="backups">Backups</h1>
|
||||
|
||||
<p>To create backups, I use <a href="https://www.borgbackup.org/">Borg</a>.
|
||||
As I keep all data on one machine, this backup process is quite simple.
|
||||
In fact, all this data is stored in a single Libvirt volume.
|
||||
To configure Borg with a simple declarative script, I use <a href="https://torsion.org/borgmatic/">Borgmatic</a>.</p>
|
||||
|
||||
<p>In order to back up the data inside the Libvirt volume, I create a snapshot to a file.
|
||||
Then I can mount this snapshot in my file system.
|
||||
The files can then be backed up while the system is still running.
|
||||
It is also possible to simply back up the Libvirt image, but this takes more time and storage <a href="https://git.kun.is/home/hypervisors/src/commit/71b96d462116e4160b6467533fc476f3deb9c306/ansible/roles/borg/backup.yml.j2"><i class="fa-solid fa-code-branch"></i></a>.</p>
|
||||
|
||||
<h1 id="monitoring-and-alerting">Monitoring and Alerting</h1>
|
||||
|
||||
<p>The last topic I would like to talk about is monitoring and alerting.
|
||||
This is something I’m still actively improving and only just set up properly.</p>
|
||||
|
||||
<h2 id="alerting">Alerting</h2>
|
||||
|
||||
<p>For alerting, I wanted something that runs entirely on my own infrastructure.
|
||||
I settled for Apprise + Ntfy.</p>
|
||||
|
||||
<p><a href="https://github.com/caronc/apprise">Apprise</a> is a server that is able to send notifications to dozens of services.
|
||||
For application developers, it is thus only necessary to implement the Apprise API to gain access to all these services.
|
||||
The Apprise API itself is also very simple.
|
||||
By using Apprise, I can also easily switch to another notification service later.
|
||||
<a href="https://ntfy.sh/">Ntfy</a> is free software made for mobile push notifications.</p>
|
||||
|
||||
<p>I use this alerting system in quite a lot of places in my infrastructure, for example when creating backups.</p>
|
||||
|
||||
<h2 id="uptime-monitoring">Uptime Monitoring</h2>
|
||||
|
||||
<p>The first monitoring setup I created, was using <a href="https://github.com/louislam/uptime-kuma">Uptime Kuma</a>.
|
||||
Uptime Kuma periodically pings a service to see whether it is still running.
|
||||
You can do a literal ping, test HTTP response codes, check database connectivity and much more.
|
||||
I use it to check whether my services and VMs are online.
|
||||
And the best part is, Uptime Kuma supports Apprise so I get push notifications on my phone whenever something goes down!</p>
|
||||
|
||||
<h2 id="metrics-and-log-monitoring">Metrics and Log Monitoring</h2>
|
||||
|
||||
<p>A new monitoring system I am still in the process of deploying is focused on metrics and logs.
|
||||
I plan on creating a separate blog post about this, so keep an eye out on that (for example using RSS :)).
|
||||
Safe to say, it is no basic ELK stack!</p>
|
||||
|
||||
<h1 id="conclusion">Conclusion</h1>
|
||||
|
||||
<p>That’s it for now!
|
||||
Hopefully I inspired someone to build something… or how not to :)</p>
|
||||
:ET
|
||||
Loading…
Add table
Add a link
Reference in a new issue