stdenv: allow for jobservers across multiple nix builds

[RaitoBezarius]

Description of changes

Retake of #143820 where I unfortunately fucked up trying to debug kernel issues and btrfs.

Original motivations by @pennae:

make -jN -lN in stdenv is a very blunt instrument. it works well when max-jobs=1, but as nix-level paralellism increases it becomes increasingly deficient. starting from a low-load situation we start max-jobs * N compilers, loadavg goes through the roof, the -lN load limit kicks in and inhibits new compilers starting until loadavg has fallen below N—at which point all make instances spawn a lot of new compilers and loadavg goes through the roof again. this oscillation leaves the system underutilized in low phases and overcommitted in high phases.

testing the current stdenv against a jobserver with 26 tokens on a 12C/24T machine shows that parallel builds of llvm_{8..11} run about 7% faster (35:52min for stdenv, 33:30min with jobserver), a larger build of llvm{5..13} is about about 11% faster (1:27h for stdenv, 1:17h with jobserver). (removing the -l from stdenv also improves utilization but is less efficient. preliminary testing here shows that -l${1.5 * N} may be a good alternative to -lN as used currently, #141266 could be a good vector to go for that instead of this whole mess. [more testing says that -l2N would be a minimum to get better utilization, but so far every -l setting we've tried has produced some underutilization except excessive large numbers like 6N or higher])

nothing in here should be regarded as a final suggestion in any way, it's more of a "hey look, this might just work". as such it's extremely rough around the edges, eg to use the jobserver the experimenter currently has to bring a /jobserver fifo filled with tokens into the nix sandbox:

nix-build -E 'with import ./. {}; pkgs.callPackage ./pkgs/os-specific/linux/nixos-jobserver {}'
touch /tmp/jobserver
sudo ./result -t 24 -g nixbld /tmp/jobserver&
nix build --extra-sandbox-paths "/jobserver=/tmp/jobserver" #...

is this something worth pursuing? a 10% speedup for hydra does seem tempting

todos before this is more generally usable:

• re-add $NIX_BUILD_CORES support (ioctl on the jobserver fd should do it)
• maybe port from cuse to fuse to not require root for everything (mmap problems may be solved by reporting jobserver file size as 0 at all times, needs testing)
• usability of tools (error messages, runtime configuration, etc)
• add support for other build systems (cargo can honor MAKEFLAGS, maybe others too)
• nixos module
• make configurable which env var(s) jscall adds jobserver information to
• metrics?
• <your suggestion here>

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = true
• Tested, as applicable:
  • Build without a job server
  • Build with a job server
  • ~Reproducibility of ISO image
• 24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

[pennae]

Suggested change

	### `runInJobServer` \<command\> \-\-\-\- \<defArgs\> \-\-\-\- \<args\> {#fun-runInJobServer}
	### `runInJobserver` \<command\> \-\-\-\- \<defArgs\> \-\-\-\- \<args\> {#fun-runInJobServer}

(multiple places)

[pennae]

running this as root is most unwise in practice but running it as another user seems to be nearly impossible. when fuse doesn't fuck us over systemd does.

[pennae]

Suggested change

	.st_mode = S_IFREG | 0660,
	.st_mode = S_IFREG | 0666,

won't work otherwise

[pennae]

Suggested change

	-j${NIX_BUILD_CORES} -l${NIX_BUILD_CORES} ---- \
	-j${NIX_BUILD_CORES} ---- \

multiple places

[pennae]

cargo apparently no longer accepts jobserver fds that aren't pipes. that's somewhat problematic.

[aviallon]

Is this work dead? It looked very promising.

[yu-re-ka]

I have some patches lying around locally to make it work with ninja, so anyone interested in picking this up, feel free to write me about it.