Add haskell.lib.incremental utility

[Gabriella439]

This adds a new incremental utility for Haskell CI that supports incremental builds based on the approach outlined in this blog post:

https://harry.garrood.me/blog/easy-incremental-haskell-ci-builds-with-ghc-9.4/

The basic idea is that instead of Nix doing a full build for a package, we split every build into two builds:

• A full build at an older point in time

  e.g. a daily or weekly time boundary

• An incremental build relative to the last full build

  This incremental build reuses the build products left over from the most recent full build.

In order to do this, though, we need a way to "snap" a package's git source input to an earlier point in time (e.g. a daily boundary or weekly boundary). This would allow multiple incremental builds to share the same full rebuild if they snap to the same time boundary.

The approach I went with to make that possible was to extend Nix's builtins.fetchGit to support a new date argument and you can find the corresponding PR for that here:

NixOS/nix#7362

That is why the incremental utility added here requires a sufficiently new version of Nix (one that would incorporate that change, presuming it is merged).

This also requires GHC 9.4 or newer in order to pick up a fix to GHC's change detection logic, as described in more detail in the above blog post.

However, if you satisfy those requirements then this works exactly the way you'd expect: all of the incremental builds only have to build the diff since the last time boundary. Moreover, if CI caches the full build then developers can also run nix build locally and only have to build the diff, too.

[Gabriella439]

This is a redo of #12659

[Gabriella439]

Note that the base branch for this is master for now instead of haskell-updates as there are a large number of build failures for us internally on the latest haskell-updates branch and I'm using our internal repository for stress testing this. Once we upgrade internally to build against the latest haskell-updates builds I'll switch this PR's base branch

[roberth]

On first reading, I thought this flag was going to be about sdist.
I've never understood why the directory is called dist. It is not for distributable files, although I can see a distant connection.

So for the output name, which I believe we should try to standardize beyond just haskellPackages, I think we should find a distinct name that isn't as distorted, so as not to disturb users who expect "distributable" files in such an output. #167670 uses a derivation name suffix incrementalBuildArtifacts rather than an output name, but it is rather distended. Perhaps that name could be distilled into something shorter.

(I hope my use of dist- words was not too distracting.)

[Gabriella439]

I think incrementalBuildArtifacts can definitely be shortened to at least buildArtifacts (since they don't necessarily need to be used for incremental builds) and it's still an accurate name. The question is if that could be shortened even further to just build or artifacts.

I would personally try to get away with shortening that to just artifacts since I believe that name has the right connotation: "artifacts" tends to mean historic byproducts that were ancillary to the final outcome (in both a programming context and non-programming context). Also, the build in buildArtifacts seems superfluous since everything produced by a Nix derivation is part of a build of some sort.

[roberth]

I'm used to artifacts referring to what's typically already in the outputs, rather than the intermediate files that do not need to be exposed for a package to be useful.
How about intermediate?

historic byproducts that were ancillary to the final outcome

Isn't this true of all software, no matter how complete?

the build in buildArtifacts seems superfluous [...]

It may refer to buildPhase, but that's not obvious.

[Gabriella439]

Yeah, I like intermediate. Or maybe intermediates (as a contraction of intermediateFiles)

[roberth]

Is it possible to "merge" this feature into #167670?

Having multiple interfaces to achieve the same thing would be bad.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixpkgs-support-for-incremental-haskell-builds/24115/2

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixpkgs-support-for-incremental-haskell-builds/24115/7

[9999years]

NB: Some of the changes from this PR were split out into #227985 and merged there.

[anka-213]

I wonder if you could hack this using IFD without needing NixOS/nix#7362? If you have a derivation that builds a fetchGit expression, I believe that as long as it generates the same output, the derivation that imports it will get the same hash and so not need to be rebuilt.

---

For example, you could do something like this:

 let rev = import (runCommand "hello.nix" {} ''mkdir $out;rev=$(${git}/bin/git --bare --git-dir=${./.git} log '--before={'$(${coreutils}/bin/date --iso-8601 -u)'T00:00Z}' --format=%H -n1) ; echo '"'$rev'"' > $out/default.nix'')
 in fetchGit { url = ./.; rev= rev;}

Now, this is absolutely horrible and very impure. It also copies the whole .git directory to the store every time you run it, but it seems to work (as long as you have the repo checked out in the current directory). It would probably be better to give git a url to clone from, but that would still require a semi-shallow clone (e.g. with git clone --filter=tree:0) each time you run it, with no way to cache the operation.

It was worse than I first thought, since I had forgotten that derivations don't have any access to git info (since that would be impure and cause needless recompilation).

[Gabriella439]

@anka-213: Maybe, but what's the point of doing this in a hacky way when it could be done in a principled way? There is already a PR that implements the principled way with tests and everything

[pwm]

@Gabriella439 what's the relation of this PR to #227985 which got merged? Do I need both for incremental builds? Thanks in advance, I'm keen to have our CI build incrementally.

[Gabriella439]

Yes, you need both PRs to support incremental builds, but this PR is blocked on NixOS/nix#7362 being merged upstream