nixos-servers

Nix definitions to configure our servers at home.

Acknowledgements

• deploy-rs: NixOS deploy tool with rollback functionality
• disko: declarative disk partitioning
• dns.nix: A Nix DSL for defining DNS zones
• flake-utils: Handy utilities to develop Nix flakes
• nixos-hardware: Hardware-specific NixOS modules. Doing the heavy lifting for our Raspberry Pi
• kubenix: declare and deploy Kubernetes resources using Nix
• nixhelm: Nix-digestible Helm charts
• sops-nix: Sops secret management for Nix

NixOS

Prerequisites

1. Install the Nix package manager or NixOS (link)
2. Enable flake and nix commands (link)

Bootstrapping

We bootstrap our servers using nixos-anywhere. This reformats the hard disk of the server and installs a fresh NixOS. Additionally, it deploys an age identity, which is later used for decrypting secrets.

⚠️ This will wipe your server completely ⚠️

1. Make sure you can decrypt the Sops-encrypted secrets in secrets/. You can test this by running sops -d secrets/serverKeys.yaml.
2. Ensure you have root SSH access to the server.
3. Run nixos-anywhere: nix run '.#bootstrap' <servername> <hostname>

Deployment

To deploy all servers at once: nix run 'nixpkgs#deploy-rs' -- '.#' -k To deploy only one server: nix run 'nixpkgs#deploy-rs' -- -k --targets '.#<host>'

Kubernetes

Prerequisites

To deploy to the Kubernetes cluster, first make sure you have an admin account on the cluster. You can generate this using nix run '.#gen-k3s-cert' <username> <servername> ~/.kube, assuming you have SSH access to the master node. This puts a private key, signed certificate and a kubeconfig in the kubeconfig directory

Bootstrapping

We are now ready to deploy to the Kubernetes cluster. Deployments are done through an experimental Kubernetes feature called ApplySets. Each applyset is responsible for a set number of resources within a namespace.

If the cluster has not been initialized yet, we must bootstrap it first. Run these deployments:

• nix run '.#bootstrap-default'
• nix run '.#bootstrap-kube-system'

Deployment

Now the cluster has been initialized and we can deploy applications. To explore which applications we can deploy, run nix flake show. Then, for each application, run nix run '.#<application>'. Or, if you're lazy: nix flake show --json | jq -r '.packages."x86_64-linux"|keys[]' | grep -- -deploy | xargs -I{} nix run ".#{}".

Known bugs

Rsync not available during bootstrap

The rsync command was removed from recent NixOS ISO which causes nixos-anywhere to fail when copying extra files. See this issue. Solution is to execute nix-env -iA nixos.rsync on the host.