Improve about page
Remove cache and compiled site
This commit is contained in:
parent
77d432ad8b
commit
03cf79e8f0
82 changed files with 321 additions and 1405 deletions
|
|
@ -0,0 +1,66 @@
|
|||
I" <p>Previously, I have used <a href="https://github.com/prometheus/node_exporter">Prometheus’ node_exporter</a> to monitor the memory usage of my servers.
|
||||
However, I am currently in the process of moving away from Prometheus to a new Monioring stack.
|
||||
While I understand the advantages, I felt like Prometheus’ pull architecture does not scale nicely.
|
||||
Everytime I spin up a new machine, I would have to centrally change Prometheus’ configuration in order for it to query the new server.</p>
|
||||
|
||||
<p>In order to collect metrics from my servers, I am now using <a href="https://fluentbit.io/">Fluent Bit</a>.
|
||||
I love Fluent Bit’s way of configuration which I can easily express as code and automate, its focus on effiency and being vendor agnostic.
|
||||
However, I have stumbled upon one, in my opinion, big issue with Fluent Bit: its <code class="language-plaintext highlighter-rouge">mem</code> plugin to monitor memory usage is <em>completely</em> useless.
|
||||
In this post I will go over the problem and my temporary solution.</p>
|
||||
|
||||
<h1 id="the-problem-with-fluent-bits-mem-plugin">The Problem with Fluent Bit’s <code class="language-plaintext highlighter-rouge">mem</code> Plugin</h1>
|
||||
|
||||
<p>As can be seen in <a href="https://docs.fluentbit.io/manual/pipeline/inputs/memory-metrics">the documentation</a>, Fluent Bit’s <code class="language-plaintext highlighter-rouge">mem</code> input plugin exposes a few metrics regarding memory usage which should be self-explaining: <code class="language-plaintext highlighter-rouge">Mem.total</code>, <code class="language-plaintext highlighter-rouge">Mem.used</code>, <code class="language-plaintext highlighter-rouge">Mem.free</code>, <code class="language-plaintext highlighter-rouge">Swap.total</code>, <code class="language-plaintext highlighter-rouge">Swap.used</code> and <code class="language-plaintext highlighter-rouge">Swap.free</code>.
|
||||
The problem is that <code class="language-plaintext highlighter-rouge">Mem.used</code> and <code class="language-plaintext highlighter-rouge">Mem.free</code> do not accurately reflect the machine’s actual memory usage.
|
||||
This is because these metrics include caches and buffers, which can be reclaimed by other processes if needed.
|
||||
Most tools reporting memory usage therefore include an additional metric that specifices the memory <em>available</em> on the system.
|
||||
For example, the command <code class="language-plaintext highlighter-rouge">free -m</code> reports the following data on my laptop:</p>
|
||||
<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code> total used free shared buff/cache available
|
||||
Mem: 15864 3728 7334 518 5647 12136
|
||||
Swap: 2383 663 1720
|
||||
</code></pre></div></div>
|
||||
|
||||
<p>Notice that the <code class="language-plaintext highlighter-rouge">available</code> memory is more than <code class="language-plaintext highlighter-rouge">free</code> memory.</p>
|
||||
|
||||
<p>While the issue is known (see <a href="https://github.com/fluent/fluent-bit/pull/3092">this</a> and <a href="https://github.com/fluent/fluent-bit/pull/5237">this</a> link), it is unfortunately not yet fixed.</p>
|
||||
|
||||
<h1 id="a-temporary-solution">A Temporary Solution</h1>
|
||||
|
||||
<p>The issues I linked previously provide stand-alone plugins that fix the problem, which will hopefully be merged in the official project at some point.
|
||||
However, I didn’t want to install another plugin so I used Fluent Bit’s <code class="language-plaintext highlighter-rouge">exec</code> input plugin and the <code class="language-plaintext highlighter-rouge">free</code> Linux command to query memory usage like so:</p>
|
||||
<div class="language-conf highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[<span class="n">INPUT</span>]
|
||||
<span class="n">Name</span> <span class="n">exec</span>
|
||||
<span class="n">Tag</span> <span class="n">memory</span>
|
||||
<span class="n">Command</span> <span class="n">free</span> -<span class="n">m</span> | <span class="n">tail</span> -<span class="m">2</span> | <span class="n">tr</span> <span class="s1">'\n'</span> <span class="s1">' '</span>
|
||||
<span class="n">Interval_Sec</span> <span class="m">1</span>
|
||||
</code></pre></div></div>
|
||||
|
||||
<p>To interpret the command’s output, I created the following filter:</p>
|
||||
<div class="language-conf highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[<span class="n">FILTER</span>]
|
||||
<span class="n">Name</span> <span class="n">parser</span>
|
||||
<span class="n">Match</span> <span class="n">memory</span>
|
||||
<span class="n">Key_Name</span> <span class="n">exec</span>
|
||||
<span class="n">Parser</span> <span class="n">free</span>
|
||||
</code></pre></div></div>
|
||||
|
||||
<p>Lastly, I created the following parser (warning: regex shitcode incoming):</p>
|
||||
<div class="language-conf highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[<span class="n">PARSER</span>]
|
||||
<span class="n">Name</span> <span class="n">free</span>
|
||||
<span class="n">Format</span> <span class="n">regex</span>
|
||||
<span class="n">Regex</span> ^<span class="n">Mem</span>:\<span class="n">s</span>+(?<<span class="n">mem_total</span>>\<span class="n">d</span>+)\<span class="n">s</span>+(?<<span class="n">mem_used</span>>\<span class="n">d</span>+)\<span class="n">s</span>+(?<<span class="n">mem_free</span>>\<span class="n">d</span>+)\<span class="n">s</span>+(?<<span class="n">mem_shared</span>>\<span class="n">d</span>+)\<span class="n">s</span>+(?<<span class="n">mem_buff_cache</span>>\<span class="n">d</span>+)\<span class="n">s</span>+(?<<span class="n">mem_available</span>>\<span class="n">d</span>+) <span class="n">Swap</span>:\<span class="n">s</span>+(?<<span class="n">swap_total</span>>\<span class="n">d</span>+)\<span class="n">s</span>+(?<<span class="n">swap_used</span>>\<span class="n">d</span>+)\<span class="n">s</span>+(?<<span class="n">swap_free</span>>\<span class="n">d</span>+)
|
||||
<span class="n">Types</span> <span class="n">mem_total</span>:<span class="n">integer</span> <span class="n">mem_used</span>:<span class="n">integer</span> <span class="n">mem_free</span>:<span class="n">integer</span> <span class="n">mem_shared</span>:<span class="n">integer</span> <span class="n">mem_buff_cache</span>:<span class="n">integer</span> <span class="n">mem_available</span>:<span class="n">integer</span> <span class="n">swap_total</span>:<span class="n">integer</span> <span class="n">swap_used</span>:<span class="n">integer</span>
|
||||
</code></pre></div></div>
|
||||
|
||||
<p>With this configuration, you can use the <code class="language-plaintext highlighter-rouge">mem_available</code> metric to get accurate memory usage in Fluent Bit.</p>
|
||||
|
||||
<h1 id="conclusion">Conclusion</h1>
|
||||
|
||||
<p>Let’s hope Fluent Bit’s <code class="language-plaintext highlighter-rouge">mem</code> input plugin is improved upon soon so this hacky solution is not needed.
|
||||
I also intend to document my new monitoring pipeline, which at the moment consists of:</p>
|
||||
<ul>
|
||||
<li>Fluent Bit</li>
|
||||
<li>Fluentd</li>
|
||||
<li>Elasticsearch</li>
|
||||
<li>Grafana</li>
|
||||
</ul>
|
||||
:ET
|
||||
Loading…
Add table
Add a link
Reference in a new issue