diff --git a/src/_posts/i-tried-forgejo-actions/2024-05-03-i-tried-forgejo-actions.md b/src/_posts/i-tried-forgejo-actions/2024-05-03-i-tried-forgejo-actions.md
new file mode 100644
index 0000000..038c878
--- /dev/null
+++ b/src/_posts/i-tried-forgejo-actions/2024-05-03-i-tried-forgejo-actions.md
@@ -0,0 +1,143 @@
+---
+layout: post
+title:  "Notes on Forgejo Actions"
+date:   2024-05-03 19:12:00 Europe/Amsterdam
+categories: forgejo actions ci act nix nixos kubernetes
+---
+
+I recently revived this blog, which is now running as a pod in my Kubernetes cluster.
+In order to deploy a new version, I had to (manualy) perform the following steps:
+
+1. Build static website from the Jekyll source
+2. Build container image using the static website
+3. Push container image to my container registry
+4. Update Kubernetes deployment manifest to use the new container image
+5. Apply the new manifest to the running blog deployment
+
+This quickly gets annoying and I started looking for a Continuous Integration (CI) system to automate this process.
+I am using [Forgejo](https://forgejo.org/) as my git server, which recently had its own CI system, Forgejo Actions, [graduated](https://forgejo.org/2023-07-release-v1201-0/) to alpha status.
+Even though this is still alpha software, I decided to try it out for myself.
+
+# Setting up a Forgejo Actions runner
+
+[Forgejo Actions](https://forgejo.org/docs/v1.20/user/actions/) is forked from [Gitea Actions](https://docs.gitea.com/usage/actions/overview) which in turn is forked from [act](https://github.com/nektos/act).
+Act is project to run GitHub Actions locally, which is the reason why Forgejo Actions are quite similar to Github Actions.
+In fact, there are quite a few references to GitHub still.
+
+Forgejo Actions work roughly as follows:
+1. You install a runner, which is a server that accepts workloads from Forgejo Actions.
+2. You register this runner with Forgejo.
+3. You define workflows on a git repository that execute steps.
+4. These workflows are submitted to a runner.
+5. Either a Docker container or an LXC container is spinned up to run the workflow to completion.
+6. The result is communicated back to Forgejo.
+
+All my workloads run on Kubernetes, and wanted the Forgejo runner to run on Kubernetes as well.
+Unfortunately, there is no Kubernetes-native way of running this runner, like for example [GitLab](https://docs.gitlab.com/runner/install/operator.html) does.
+However, there is [an example deployment](https://code.forgejo.org/forgejo/runner/src/branch/main/examples/kubernetes) using a Docker-in-Docker setup.
+This does however require to run the image as a privileged container, and (spoiler) this setup seems to be very unreliable.
+
+# Using Forgejo Actions to build Nix derivations
+
+Both the static website and container image for my blog are built using Nix (which I wrote about [here]({% post_url nix-jekyll-derivation/2024-05-03-nix-jekyll-derivation.md %})).
+Therefore, it made sense to me to use the [official Docker image](https://hub.docker.com/r/nixos/nix).
+However, it seems Forgeo Actions has an undocumented dependency on `/bin/sleep` being present inside the container image.
+It seems sleep is being used to check whether the image is working correctly.
+Nix being very unconventional, does not have `/bin/sleep` by default in its Docker image.
+Therefore, I extended the offical Docker image using Nix below.
+The `coreutils` package contains the `sleep` binary which is linked inside the image using `pathsToLink`.
+
+```nix
+let
+  nixFromDockerHub = pkgs.dockerTools.pullImage {
+    imageName = "nixos/nix";
+    imageDigest = "sha256:b3dc72ab3216606d52357ee46f0830a0cc32f3e50e00bd490efa1a8304e9f99d";
+    sha256 = "sha256-FvDlbSnCmPtWTn4eG3hu8WVK1Wm3RSi2T+CdmIDLkG4=";
+    finalImageTag = "2.22.0";
+    finalImageName = "nix";
+  };
+  };
+in
+{
+  packages.forgejo-nix-action = pkgs.dockerTools.buildImage {
+    name = "forgejo-nix-action";
+    tag = "latest";
+    fromImage = nixFromDockerHub;
+
+    copyToRoot = pkgs.buildEnv {
+      name = "image-root";
+      paths = with pkgs; [ coreutils ];
+      pathsToLink = [ "/bin" ];
+    };
+  };
+}
+```
+
+This is sufficient to be able to run `nix build` commands inside Forgejo Action, for example:
+```yaml
+on: [ push ]
+jobs:
+  blog-pim:
+    runs-on: docker
+    container:
+      image: git.kun.is/home/forgejo-nix-action:687d16c49ea7936068bac64ec68c480a9d681962
+    steps:
+      - name: Clone repository
+        run: git clone ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}.git src
+      - name: Build image
+        run: nix build --out-link image ./src#packages.x86_64-linux.container-image
+```
+
+This workflows clones the source code of my blog, then builds a container image for it using Nix.
+
+# Pushing the image to a container registry
+
+Being able to automatically build container images is useless if we don't publish them somewhere.
+[Skopeo](https://github.com/containers/skopeo) is a real life-saver here, and I would strongly recommend this tool when interacting with container images.
+Before settling with Skopeo, I first tried using just Docker.
+However, for many operations, Docker needs a running Docker daemon and this is annoying to setup inside an already annoying Docker-in-Docker setup.
+Another option I explored was Podman but it was not even able to read the container images built by Nix.
+
+These are the steps I ended up with to push a container image to a registry using Skopeo:
+```yaml
+- name: Log into container registry
+  run: /bin/skopeo login --tls-verify --username ${{ vars.RUNNER_USER }} --password ${{ secrets.RUNNER_TOKEN }} ${GITHUB_SERVER_URL}
+- name: Push image to container registry
+  run: |
+    /bin/skopeo --insecure-policy copy docker-archive:image docker://${GITHUB_SERVER_URL#https://}/${GITHUB_REPOSITORY_OWNER}/blog-pim:latest
+```
+
+# Failing to deploy to Kubernetes
+
+Having pushed a new container image to a registry, we just have to updated the Kubernetes deployment to use the new image.
+This is unfortunately something I have not been able to get working.
+For some reason, anytime I run `nix run` inside a step, the k3s systemd service crashes.
+It seems during the execution of `nix run`, `etcd` is continuously timing out.
+After a few seconds of this, k3s deems it unhealthy and restarts itself.
+
+I have a hunch running `nix run` puts so much strain on the disk that `etcd` is not able to quickly write to disk.
+It is known that `etcd` needs very quick disk times to function correctly.
+I have however not been able to reproduce this issue outside of the container.
+This leads me to believe that perhaps the Docker-in-Docker setup is the cause of this issue.
+
+# Conclusions
+
+I think Forgejo Actions really needs a first-class Kubernetes treatment.
+It seems to me having Action runners on Kubernetes gives free improvements in terms of scalability and management.
+Going further, I will not be using Forgejo Actions for CI, but will explore some systems that run on Kubernetes natively.
+In particular, I'm interested in [Argo Workflows](https://argo-workflows.readthedocs.io/en/latest/), and perhaps I will give [GitLab](https://docs.gitlab.com/runner/install/operator.html) a try.
+
+However in general, and this is not a critique on Forgejo Actions in particular, I don't fully agree with the way these "Actions" systems work.
+It seems to leading way to use these actions is to use a `runs-on` directive and specify some container image, perhaps the latest Ubuntu version (who thought it was a good idea to turn OSes into images?).
+Then, people install some tools using the image's package manager after which their actual jobs runs.
+Unless they are very well maintained, these actions will probably stop working after a while when the underlying container image is discontinued from support.
+Or perhaps a package is updated, which is incompatable with the action.
+Or there is some hidden dependency that changes in the underlying container image.
+Or ...
+
+I believe Nix could be an amazing tool to improve these "Actions" systems.
+Using Nix, you wouldn't need a bloated container image with lots of tools you won't need.
+You could take advantage of Nixpkgs and install only the tools you need to run your job.
+Everything inside the image would be version-pinned and reproducible and will continue to work in 50 years.
+I'm not entirely sure how such a system would even look, but my gut tells me it should be possible.
+I'm unfortunately not aware whether such a system currently exists though.
diff --git a/src/_posts/nix-jekyll-derivation/2024-05-03-nix-jekyll-derivation.md b/src/_posts/nix-jekyll-derivation/2024-05-03-nix-jekyll-derivation.md
index 3741c32..68bfb64 100644
--- a/src/_posts/nix-jekyll-derivation/2024-05-03-nix-jekyll-derivation.md
+++ b/src/_posts/nix-jekyll-derivation/2024-05-03-nix-jekyll-derivation.md
@@ -1,7 +1,7 @@
 ---
 layout: post
 title:  "Building a Jekyll Static Website using Nix"
-date:   2024-05-02 23:51:00 Europe/Amsterdam
+date:   2024-05-03 15:44:00 Europe/Amsterdam
 categories: jekyll nix blog ruby
 ---