nixcon-2024-presentation-public/presentation.md

marp	theme	paginate	footer
true	gaia	true	October 26th, 2024

Kubenix: Leveraging NixOS modules to generate Kubernetes manifests

Pim Kunis

---

Agenda

1. My background
2. Kubernetes
3. Kubenix

---

$ whoami

• Pim Kunis
  • Matrix: @pim:envs.net
  • GitHub: @pizzapim
• Consultant @ Sue
  • Currently doing Linux + Ansible stuff
• Relatively new to Nix
• Avid self-hoster

---

Self-hosting

---

Self-hosting

3 mini PCs (Gigabyte Brix with Intel Celeron J4105)

---

NixOS for a cluster?

---

NixOS for a cluster?

• Balance services
• Maintenance
• Services are tied to a particular host
  • Pets vs cattle

---

Agenda

1. My background
2. Kubernetes
3. Kubenix

---

Enter: Kubernetes

• Abstracts infrastructure
• Container orchestration
  • Workloads are containers (pods)
• Distributed state (etcd)
  • Actual state
  • Desired state
  • Reconciliation

---

Kubernetes

---

Kubernetes

---

Kubernetes

---

Mutating state in Kubernetes

---

Mutating state in Kubernetes

---

Mutating state in Kubernetes

---

Mutating state in Kubernetes

---

Helm

• Package manager
• Templating YAML
• Cursed

---

Helm

---

Helm

---

Helm

---

Agenda

1. My background
2. Kubernetes
3. Kubenix

---

Kubenix ✨

• Generate manifests using NixOS modules!
• Benefits:
  • No ugly YAML templates
  • Functional language
  • Type and structure checking
  • Modularity

---

How Kubenix works

1. My background
2. Kubernetes
3. Kubenix
   • OpenAPI specification
   • Generate NixOS module
   • Use generated module
   • Generate manifest

---

The OpenAPI spec

• Standard for describing APIs
• YAML or JSON
• Specifies how to interact with Kubernetes API

---

The OpenAPI spec

---

The OpenAPI spec

---

The OpenAPI spec

---

How Kubenix works

1. My background
2. Kubernetes
3. Kubenix
   • OpenAPI specification
   • Generate NixOS module
   • Use generated module
   • Generate manifest

---

Generating the NixOS module

• Generate Nix code
  • Templating?

---

Generating the NixOS module

''
definitions = {
  # Here we loop over each "definition"
  ${concatStrings (mapAttrsToList (name: value: ''
  "${name}" = {
    ${optionalString (hasAttr "options" value) "
      options = {${concatStrings (mapAttrsToList (name: value: ''                                                            
      "${name}" = ${value};
    '')
    value.options)}};
      "}

    ${optionalString (hasAttr "config" value) ''
    config = {${concatStrings (mapAttrsToList (name: value: ''
        "${name}" = ${value};
      '')
      value.config)}};
  ''}
  };
'')
definitions)}
};
''

---

The generated NixOS module

    "io.k8s.api.core.v1.Pod" = {
      options = {
        "apiVersion" = mkOption {
          description = "...";
          type = (types.nullOr types.str);
        };
        "kind" = mkOption {
          description = "...";
          type = (types.nullOr types.str);
        };
        "metadata" = mkOption {
          description = "...";
          type = (types.nullOr (submoduleOf "io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta"));                                  
        };
        "spec" = mkOption {
          description = "...";
          type = (types.nullOr (submoduleOf "io.k8s.api.core.v1.PodSpec"));
        };
        "status" = mkOption {
          description = "...";
          type = (types.nullOr (submoduleOf "io.k8s.api.core.v1.PodStatus"));
        };
      };
    };

---

How Kubenix works

1. My background
2. Kubernetes
3. Kubenix
   • OpenAPI specification
   • Generate NixOS module
   • Use generated module
   • Generate manifest

---

Using the module

---

Using the module

---

Using the module

---

Using the module

---

Using the module

---

Example of type checking

{
    kubernetes.resources.pods.example-pod = {
        spec.containers.jellyfin-container = {
            image = "jellyfin/jellyfin:latest";
            ports = [{ containerPort = "foo"; }];
        };
    };
};

 error: A definition for option `kubernetes.(...).containerPort' is not of type `signed integer'.

---

An undefined option

{
    kubernetes.resources = {
        foo.bar = "baz";
    }
}

error: The option `kubernetes.api.resources.foo' does not exist.

---

How Kubenix works

1. My background
2. Kubernetes
3. Kubenix
   • OpenAPI specification
   • Generate NixOS module
   • Use generated module
   • Generate manifest

---

kubenix.evalModules

{
  inputs.kubenix.url = "github:hall/kubenix";
  outputs = {self, kubenix, ... }@inputs: let
    system = "x86_64-linux";
  in {
    packages.${system}.default = (kubenix.evalModules.${system} {
      module = { kubenix, ... }: {
        imports = [ kubenix.modules.k8s ];
        kubernetes.resources.pods.example.spec.containers.jellyfin.image = "jellyfin/jellyfin:latest";
      };
    }).config.kubernetes.result;
  };
}

---

kubenix.evalModules

{
  "apiVersion": "v1",
  "items": [
    {
      "apiVersion": "v1",
      "kind": "Pod",
      "metadata": {
        "annotations": {
          "kubenix/k8s-version": "1.30",
          "kubenix/project-name": "kubenix"
        },
        "labels": {
          "kubenix/hash": "84130bb6a26b9066eb2bdcaf2e68148a6e0648b0"                                                                                                                  
        },
        "name": "example"
      },
      "spec": {
        "containers": [
          {
            "image": "jellyfin/jellyfin:latest",
            "name": "jellyfin"
          }
        ]
      }
    }
  ],
  "kind": "List",
  "labels": {
    "kubenix/hash": "84130bb6a26b9066eb2bdcaf2e68148a6e0648b0",
    "kubenix/k8s-version": "1.30",
    "kubenix/project-name": "kubenix"
  }
}

---

Summary

• Kubenix generates Kubernetes manifests
• Converts OpenAPI spec to a NixOS module
• Build Kubernetes deployments with:
  • No ugly YAML templates
  • Modularity
  • In a functional language
  • Type and structure checking

---

Terraform: OpenAPI Provider Spec Generator

---

Bonus: nix-snapshotter

• Images are in Nix store
• Fully declarative

---

Bonus: NixNG

• NixOS in a container
• init-system independent