pim/blog
2
0
Fork 1
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

Improve about page

Browse source 
Remove cache and compiled site
This commit is contained in:
Pim Kunis 2024-04-30 13:12:09 +02:00
parent 77d432ad8b
commit 03cf79e8f0
82 changed files with 321 additions and 1405 deletions
Download patch file Download diff file Expand all files Collapse all files

51
.jekyll-cache/Jekyll/Cache/Jekyll--Converters--Markdown/dd/fa550314d3f6e485fefcd71068b25447db3acbd8d5f496b19d502161a999cd Normal file
View file

@ -0,0 +1,51 @@
I"ë<p><a href="https://borgbackup.readthedocs.io/en/stable/">BorgBackup</a> and <a href="https://torsion.org/borgmatic/">Borgmatic</a> have been my go-to tools to create backups for my home lab since I started creating backups.
Using <a href="https://wiki.archlinux.org/title/systemd/Timers">Systemd Timers</a>, I regularly create a backup every night.
I also monitor successful execution of the backup process, in case some error occurs.
However, the way I set this up resulted in not receiving notifications.
Even though it boils down to RTFM, I’d like to explain my error and how to handle errors correctly.</p>
<p>I was using the <code class="language-plaintext highlighter-rouge">on_error</code> option to handle errors, like so:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">on_error</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s1">'</span><span class="s">apprise</span><span class="nv"> </span><span class="s">--body="Error</span><span class="nv"> </span><span class="s">while</span><span class="nv"> </span><span class="s">performing</span><span class="nv"> </span><span class="s">backup"</span><span class="nv"> </span><span class="s">&lt;URL&gt;</span><span class="nv"> </span><span class="s">||</span><span class="nv"> </span><span class="s">true'</span>
</code></pre></div></div>
<p>However, <code class="language-plaintext highlighter-rouge">on_error</code> does not handle errors from the execution of <code class="language-plaintext highlighter-rouge">before_everything</code> and <code class="language-plaintext highlighter-rouge">after_everything</code> hooks.
My solution to this was moving the error handling up to the Systemd service that calls Borgmatic.
This results in the following Systemd service:</p>
<div class="language-systemd highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">[Unit]</span>
<span class="nt">Description</span><span class="p">=</span>Backup data using Borgmatic
<span class="c"># Added</span>
<span class="nt">OnFailure</span><span class="p">=</span>backup-failure.service
<span class="k">[Service]</span>
<span class="nt">ExecStart</span><span class="p">=</span>/usr/bin/borgmatic --config /root/backup.yml
<span class="nt">Type</span><span class="p">=</span>oneshot
</code></pre></div></div>
<p>This handles any error, be it from Borgmatic’s hooks or itself.
The <code class="language-plaintext highlighter-rouge">backup-failure</code> service is very simple, and just calls Apprise to send a notification:</p>
<div class="language-systemd highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">[Unit]</span>
<span class="nt">Description</span><span class="p">=</span>Send backup failure notification
<span class="k">[Service]</span>
<span class="nt">Type</span><span class="p">=</span>oneshot
<span class="nt">ExecStart</span><span class="p">=</span>apprise --body="Failed to create backup!" &lt;URL&gt;
<span class="k">[Install]</span>
<span class="nt">WantedBy</span><span class="p">=</span>multi-user.target
</code></pre></div></div>
<h1 id="the-aftermath-or-what-i-learned">The Aftermath (or what I learned)</h1>
<p>Because the error handling and alerting weren’t working propertly, my backups didn’t succeed for two weeks straight.
And, of course, you only notice your backups aren’t working when you actually need them.
This is exactly what happened: my disk was full and a MariaDB database crashed as a result of that.
Actually, the whole database seemed to be corrupt and I find it worrying MariaDB does not seem to be very resilient to failures (in comparison a PostgreSQL database was able to recover automatically).
I then tried to recover the data using last night’s backup, only to find out there was no such backup.
Fortunately, I had other means to recover the data so I incurred no data loss.</p>
<p>I already knew it is important to test backups, but I learned it is also important to test failures during backups!</p>
:ET