nixos-servers/docs/longhorn.md

Longhorn notes

Migration from NFS to Longhorn

1. Delete the workload, and delete the PVC and PVC using NFS.
2. Create Longhorn volumes as described below.
3. Copy NFS data from lewis.dmz to local disk.
4. Spin up a temporary pod and mount the Longhorn volume(s) in it:

    {
      pods.testje.spec = {
        containers.testje = {
          image = "nginx";
   
          volumeMounts = [
            {
              name = "uploads";
              mountPath = "/hedgedoc/public/uploads";
            }
          ];
        };
   
        volumes = {
          uploads.persistentVolumeClaim.claimName = "hedgedoc-uploads";
        };
      };
    }
   

5. Use kubectl cp to copy the data from the local disk to the pod.
6. Delete the temporary pod.
7. Create the workload with updated volume mounts.
8. Delete the data from local disk.

Creation of new Longhorn volumes

While it seems handy to use a K8s StorageClass for Longhorn, we do not want to use that. If you use a StorageClass, a PV and Longhorn volume will be automatically provisioned. These will have the name pvc-<UID of PVC>, where the UID of the PVC is random. This makes it hard to restore a backup to a Longhorn volume with the correct name.

Instead, we want to manually create the Longhorn volumes via the web UI. Then, we can create the PV and PVC as usual using our K8s provisioning tool (e.g. Kubectl/Kubenix).

Follow these actions to create a Volume:

1. Using the Longhorn web UI, create a new Longhorn volume, keeping the following in mind:
   • The size can be some more than what we expect to reasonable use. We use storage-overprovisioning, so the total size of volumes can exceed real disk size.
   • The number of replicas should be 2.
2. Create the PV, PVC and workload as usual.

Disaster recovery using Longhorn backups

Backing up Longhorn volumes is very easy, but restoring them is more tricky. We consider here the case when all our machines are wiped, and all we have left is Longhorn backups. To restore a backup, perform the following actions:

1. Restore the latest snapshot in the relevant Longhorn backup, keeping the following in mind:
   • The name should remain the same (i.e. the one chosen at Longhorn volume creation).
   • The number of replicas should be 2.
   • Disable recurring jobs.
2. Enable the "backup-nfs" recurring job for the Longhorn volume.
3. Disable the "default" recurring job group for the Longhorn volume.
4. Create the PV, PVC and workload as usual.