microvm #5

New issue

Closed

opened 2023-11-17 13:22:28 +00:00 by pim · 3 comments

pim commented

2023-11-17 13:22:28 +00:00

Owner

https://github.com/astro/microvm.nix

✅ How can we use SSH with the VM?

It seems there is no support for a direct TTY on the VM (see: https://github.com/astro/microvm.nix/issues/123).
This is not really a problem, as we can just use SSH instead (like we do with libvirt currently).

There are some options for creation a network device on the host listed at https://astro.github.io/microvm.nix/interfaces.html:

user: not sure how this works, but I assume qemu doing some form of NAT. Not a great fit, as it does not create an interface on the host.
tap: uses a virtual tuntap device on the host. This interface has to be created on the host first.
macvtap: uses the same physical network interface on the host. Definitely not ideal, because we want this interface is a vlan trunk port.
bridge: qemu creates a tuntap device and adds this to a bridge.

The bridge mode is what we are using with libvirt now. However, this does require qemu to run without "-sandbox on". Better would be to use the tap option. This does require us to create a network device on the host before creating the VM.

It also seems like we can let systemd create the network interface for us. Then, using systemd-networkd, we can automatically attach this network interface to a bridge by matching the name of it (e.g. "vm-*"). This way we don't have to create the network interface ourselves!

✅ What happens when we remove a VM declaration from the hosts' NixOS definition?

Will it get removed or will the VM become orphaned?

When we remove the VM declaration, the VM keeps running. We can stop it, and then restart it even. We should therefore be careful not to leave VMs running.

✅ Where does the OS storage live?

It seems that you can make changes to the filesystem from inside the VM while it is running, but this is only temporary: if you reboot, everything is gone. It seems the only way to get persistency is using shared storage (9p or virtiofs).

This also means that any configurations done at run-time will be removed at reboot. Therefore we can only use the OS image to make configuration changes. And the OS image can only be switched by rebooting after updating the image.

✅ Where does the guest OS image by default come from?

The kernel and initrd are both by default taken from the default nixos kernel/initrd. We could if we want to change this. All the rest is simply a nixos configuration.

What happens when we push a new NixOS definition to the VM?

It seems a new definition is created and pushed to the machine. However, the currently booted system is not recreated immediately. Instead a symlink at /var/lib/microvms/<name>/current points at the most up-to-date VM definition, while /var/lib/microvms/<name>/bootedstill points at the booted VM definition. Still not sure whether it is possible to immediately reboot the machine as well.

https://github.com/astro/microvm.nix ### ✅ How can we use SSH with the VM? It seems there is no support for a direct TTY on the VM (see: https://github.com/astro/microvm.nix/issues/123). This is not really a problem, as we can just use SSH instead (like we do with libvirt currently). There are some options for creation a network device on the host listed at https://astro.github.io/microvm.nix/interfaces.html: - user: not sure how this works, but I assume qemu doing some form of NAT. Not a great fit, as it does not create an interface on the host. - tap: uses a virtual tuntap device on the host. This interface has to be created on the host first. - macvtap: uses the same physical network interface on the host. Definitely not ideal, because we want this interface is a vlan trunk port. - bridge: qemu creates a tuntap device and adds this to a bridge. The bridge mode is what we are using with libvirt now. However, this does require qemu to run without "-sandbox on". Better would be to use the tap option. This does require us to create a network device on the host before creating the VM. It also seems like we can let systemd create the network interface for us. Then, using systemd-networkd, we can automatically attach this network interface to a bridge by matching the name of it (e.g. "vm-*"). This way we don't have to create the network interface ourselves! ### ✅ What happens when we remove a VM declaration from the hosts' NixOS definition? Will it get removed or will the VM become orphaned? When we remove the VM declaration, the VM keeps running. We can stop it, and then restart it even. We should therefore be careful not to leave VMs running. ### ✅ Where does the OS storage live? It seems that you can make changes to the filesystem from inside the VM while it is running, but this is only temporary: if you reboot, everything is gone. It seems the only way to get persistency is using shared storage (9p or virtiofs). This also means that any configurations done at run-time will be removed at reboot. Therefore we can only use the OS image to make configuration changes. And the OS image can only be switched by rebooting after updating the image. ### ✅ Where does the guest OS image by default come from? The kernel and initrd are both by default taken from the default nixos kernel/initrd. We could if we want to change this. All the rest is simply a nixos configuration. ### What happens when we push a new NixOS definition to the VM? It seems a new definition is created and pushed to the machine. However, the currently booted system is not recreated immediately. Instead a symlink at `/var/lib/microvms/<name>/current` points at the most up-to-date VM definition, while `/var/lib/microvms/<name>/booted`still points at the booted VM definition. Still not sure whether it is possible to immediately reboot the machine as well.

pim commented

2023-11-20 12:48:20 +00:00

Author

Owner

To be fixed later:
warning: system.stateVersion is not set, defaulting to 23.05. Read why this matters on https://nixos.org/manual/nixos/stable/options.html#opt-system.stateVersion.

To be fixed later: warning: system.stateVersion is not set, defaulting to 23.05. Read why this matters on https://nixos.org/manual/nixos/stable/options.html#opt-system.stateVersion.

pim commented

2023-11-21 19:26:34 +00:00

Author

Owner

It seems like we can deploy the VM's config using SSH as well, but it is quite complex:
https://gitea.c3d2.de/c3d2/nix-config/src/branch/master/packages.nix#L240

Also, we won't be able to use deploy-rs, but that is okay I think.

However, I think for now we want to go down the beaten path: require a reboot everytime the VM definition is changed.

It seems like we can deploy the VM's config using SSH as well, but it is quite complex: https://gitea.c3d2.de/c3d2/nix-config/src/branch/master/packages.nix#L240 Also, we won't be able to use deploy-rs, but that is okay I think. However, I think for now we want to go down the beaten path: require a reboot everytime the VM definition is changed.

pim commented

2023-11-22 10:56:54 +00:00

Author

Owner

Okay, I am not totally convinced this is a good solution after all.

Pros

Can define whole VM in nix, no other tools necessary
Very lightweight VMs

Cons

Have to reboot when changing any configuration on the VM
Cannot use existing tooling (deploy-rs) for deployment

Okay, I am not totally convinced this is a good solution after all. ### Pros - Can define whole VM in nix, no other tools necessary - Very lightweight VMs ### Cons - Have to reboot when changing any configuration on the VM - Cannot use existing tooling (deploy-rs) for deployment