Transparent VM builds - a hook to start and stop a builder VM

[roberth]

Is your feature request related to a problem? Please describe.

Nixpkgs now has a darwin.builder package, which is a NixOS VM that's pre-built by Hydra, so that macOS users can create their first linux builder VM using it.

However, this still requires user intervention, and it is not enabled by default, nor should it be, because a VM has non-trivial overhead. Yet, there's something Nix can do to help.

Furthermore, other startable resources that are not a VM could provide a store.

Describe the solution you'd like

A configuration similar to remote builders, but for local VM builders.
What needs to be configured

• Command that starts the VM
  • Port where the VM's sshd can be reached is printed to stdout when the port is available
• Command for shutting down the VM
• The scheduling related parameters, like max concurrent builds, supported features, mandatory features

Ideally the scheduler is aware that it's the same machine and it applies the "rule" vm.concurrentBuildLimit = min(host.jobs, vm.jobs)

The start stop start cycle is managed by a new store called LifeCycleStore that returns and/or delegates to whatever store URI/config is returned from the configured start hook. The state of the resource (including store URI of the store to delegate to and shutdown command) is stored centrally, ideally managed by the nix daemon - LocalStore.

A nix.conf with builder VMs can be generated by configuration management tools such as nix-darwin and NixOS; perhaps even the official installer?

This also makes Nix useful at an earlier stage of the porting process:

• the VM does not need an evaluator
• the host does not need a sandbox

Example:

• effort: port the evaluator to Windows
  • use case: get a nice UI for Linux packaging on Windows

Describe alternatives you've considered

Start a VM per build. This would be slow, especially for fine grained builds.

Status quo: user configures the VM, starts the VM, wasting their time and their system's memory

Additional context

darwin.builder

Priorities

Add 👍 to issues you find important.

[NickCao]

This is also a great chance to further separate the three parts of nix: evaluator, builder (or sandbox) and store. Instead of having the hook specifically designed for VMs, we can generalize it to support OCI runtimes and more. We may even reuse the OCI interface for VM invocations, e.g. kata-containers are basically VMs pretending to be containers.

[roberth]

You could build something OCI-based on top of this, so I don't think integrating OCI directly should be called a generalization. The interface proposed here is small (which is good) and it reuses already implemented Nix functionality such as remote stores via ssh, or any "store" implementation (which includes the build functionality).
OCI on the other hand exposes us to a large interface that's not necessarily a great fit, and I think images and layers are an example of that. Nonetheless OCI could be valuable, but I don't think it would replace this proposal. If you have ideas about how an OCI integration should work in more detail, feel free to open an issue.

the three parts of nix: evaluator, builder (or sandbox) and store

The evaluator is already well separated. The builder being part of the store interface isn't particularly elegant, but not a blocker when you consider that you could have a store object that refers to the same actual store but builds things differently.

Instead of having the hook specifically designed for VMs

I don't think the idea of having a store (builder) like this would be VM-specific. It supports VMs well, but you could start something over the network and refer to that instead. LifeCycleStore doesn't do much more than run the configured command, read a store url from its output and launch another command when the store (builder) is not needed anymore. I've just phrased it in terms of what I presume would be its primary use case and the problem I originally wanted to solve.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/is-nixbsd-a-posibility/29612/19