ci/eval: make eval for non-native platforms less incorrect

[emilylange]

We commonly use platform-dependent conditional patterns like lib.meta.availableOn stdenv.hostPlatform and stdenv.hostPlatform.isLinux to enable different features in a given derivation or to evaluate completely different derivations based on the platform.

For example, source builds of a given derivation may only be available on linux but not on darwin. The use of such conditionals allow us to fall back to patched binaries on darwin instead.

In chromedriver (pkgs/development/tools/selenium/chromedriver/default.nix), we use

if lib.meta.availableOn stdenv.hostPlatform chromium then
  callPackage ./source.nix { }
else
  callPackage ./binary.nix { }

To provide some context, chromedriver source builds are based on chromium.mkDerivation and chromium is limited to lib.platforms.linux. Based on the same chromium.mkDerivation, we also do source builds for electron (pkgs/top-level/all-packages.nix):

electron_33 = if lib.meta.availableOn stdenv.hostPlatform electron-source.electron_33 then electron-source.electron_33 else electron_33-bin; electron_34 = electron_34-bin;
electron = electron_34;

And finally, the top-level jdk (Java) attribute has a lot of indirection, but eventually also boils down to stdenv.hostPlatform.isLinux for source builds and binaries for x86_64-darwin and aarch64-darwin.

A surprising amount of electron and jdk consumers use variations of meta.platforms = electron.meta.platforms in their own meta block. Due to internal implementation details, the conditionals in those top-level attributes like chromedriver, electron and jdk are evaluated based on the value from builtins.currentSystem and not the system passed to import <nixpkgs> { }.

This then causes chromedriver, electron, jdk and all dependents that inherit those meta.platforms to appear only available on linux despite also being available on darwin. Hydra is affected similarly, but it's a lot more nuanced and in practice not actually that bad.

The addition of --eval-system ensures that builtins.currentSystem matches the requested platform.

As a bonus, this also fixes the store paths of an impure test that should probably be made pure:

@@ -885069,13 +886119,13 @@
     "out": "/nix/store/lb2500hc69czy4sfga9mbh2k679cr1rp-test-compressDrv"
   },
   "tests.config.allowPkgsInPermittedInsecurePackages.aarch64-darwin": {
-    "out": "/nix/store/0l5h8svrpzwymq35mnpvx82gyc7nf8s4-hello-2.12.1"
+    "out": "/nix/store/v1zjb688mp4y2132b6chii43d5kkxnpa-hello-2.12.1"
   },
   "tests.config.allowPkgsInPermittedInsecurePackages.aarch64-linux": {
-    "out": "/nix/store/0l5h8svrpzwymq35mnpvx82gyc7nf8s4-hello-2.12.1"
+    "out": "/nix/store/hb21z2zdk03dwygsw5lvpa8zc3fbr500-hello-2.12.1"
   },
   "tests.config.allowPkgsInPermittedInsecurePackages.x86_64-darwin": {
-    "out": "/nix/store/0l5h8svrpzwymq35mnpvx82gyc7nf8s4-hello-2.12.1"
+    "out": "/nix/store/gljdqsf0mxv1j8zb04phx9ws09pp7z3l-hello-2.12.1"
   },
   "tests.config.allowPkgsInPermittedInsecurePackages.x86_64-linux": {
     "out": "/nix/store/0l5h8svrpzwymq35mnpvx82gyc7nf8s4-hello-2.12.1"

Diff stats between two full evals based on 75c8548 with and without this fix on x86_64-linux:

# git diff --no-index --stat /nix/store/659l3xp78255wx7abbahggsnrlj3a1la-combined-result/outpaths.json /nix/store/4fhlq4g5qa65cxbibskq9pma40zigrx7-combined-result/outpaths.json
 /nix/store/{659l3xp78255wx7abbahggsnrlj3a1la-combined-result => 4fhlq4g5qa65cxbibskq9pma40zigrx7-combined-result}/outpaths.json | 1416 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 1405 insertions(+), 11 deletions(-)

The full diff is available as a gist at https://gist.github.com/emilylange/d40c50031fc332bbcca133ad56d224f6.

When we added electron_34 only as binary instead of the usual source on linux with binary fallback in #376770 (cfed9a1) and made the unversioned electron top-level point to the newly added electron_34 instead of electron_33, the GitHub workflow suddenly reported 20 new packages. Of those 20 reported packages, 17 where false-positives caused by dropping the wrongly evaluated conditional.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 25.05 Release Notes (or backporting 24.11 and 25.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

[wolfgangwalther]

Seems reasonable to me.

[wolfgangwalther]

Due to internal implementation details, the conditionals in those top-level attributes like chromedriver, electron and jdk are evaluated based on the value from builtins.currentSystem and not the system passed to import <nixpkgs> { }.

Could you point me to where that happens? I can't find it.