Using libgit for deduplication yields drastically worse eval performance

[roberth]

Originally posted by @erikarvstedt in #9485 (comment)

Using libgit for deduplication yields a very simple implementation but drastically worsens eval performance.

I've mentioned this before in the "Source tree abstraction" PR.
A current benchmark confirms these results:
Eval'ing the paperless NixOS test with the libgit accessor (PR "Source tree abstraction") takes 60% longer compared to the branch the PR is based on.

Benchmark source

# https://github.com/NixOS/nix/pull/6530 as of 2023-12-31
nix build github:edolstra/nix/2055e28aedb3918d91dcd80a5cf9bdd32242c822 -o /tmp/nix/git-accessor
# The version of master which the above PR is based on
nix build github:nixos/nix/9dbfd186b129ddee4a7a66958be9abdf6dd6a668 -o /tmp/nix/baseline

nix_eval() {
    nix=$1
    shift
    # A test with a single VM system evaluation that accesses lots of files
    # nixpkgs-unstable as of 2023-02-01
    /tmp/nix/$nix/bin/nix eval --json github:nixos/nixpkgs/ea692c2ad1afd6384e171eabef4f0887d2b882d3#nixosTests.paperless "$@"
}
export -f nix_eval

hyperfine --warmup 3 \
          'nix_eval baseline --read-only' \
          'nix_eval git-accessor --read-only'

rm -rf /tmp/nix

# Benchmark 1: nix_eval baseline --read-only
#   Time (mean ± σ):      3.595 s ±  0.042 s    [User: 3.221 s, System: 0.368 s]
#   Range (min … max):    3.554 s …  3.684 s    10 runs
#  
# Benchmark 2: nix_eval git-accessor --read-only
#   Time (mean ± σ):      5.786 s ±  0.059 s    [User: 5.273 s, System: 0.460 s]
#   Range (min … max):    5.712 s …  5.909 s    10 runs
#  
# Summary
#   'nix_eval baseline --read-only' ran
#     1.61 ± 0.02 times faster than 'nix_eval git-accessor --read-only'

This issue can also be confirmed with this PR:
Copying a github input (which is already present in the git tarball cache) to the store takes over 3x longer than extracting an already downloaded tarball to the store with Nix master.

Also, extracting tarballs to the libgit cache is very slow. Evaluating a NixOS system flake that has multiple github inputs with cold caches can be 5x slower compared to Nix master, moving the initial eval time into the minutes territory.

Eval perf is already a major pain point of Nix. Adding the libgit penalty makes Nix impractical to use for larger NixOS systems.
I'm sure most users would prefer the existing disk space overhead to heavily decreased perf.
For specific use cases the libgit cache might be useful, so it could be made available via a Nix config option.

[erikarvstedt]

Here's a benchmark of a common usage pattern from the opposite end of the spectrum:
Eval a single pkg from a nixpkgs github input that hasn't been downloaded yet:

nix eval nixpkgs#qemu

Benchmark source

# https://github.com/NixOS/nix/pull/6530 as of 2023-12-31
nix build github:edolstra/nix/2055e28aedb3918d91dcd80a5cf9bdd32242c822 -o /tmp/nix/git-accessor
# The version of master which the above PR is based on
nix build github:nixos/nix/9dbfd186b129ddee4a7a66958be9abdf6dd6a668 -o /tmp/nix/baseline
# The last commit of https://github.com/NixOS/nix/pull/6530 that uses zipballs for github inputs
nix build github:nixos/nix/4072024d79b088739ab3b38eb4965ae7d5967d22 -o /tmp/nix/zipball

nix_eval() {
    nix=$1
    shift
    # `nix/zipball` can't eval nixpkgs >= 23.11
    nixos_23_05=nixos/nixpkgs/4e98de01b793b087cbce32e4c982ae2cdd82978a
    XDG_CACHE_HOME=/tmp/nix-cache /tmp/nix/$nix/bin/nix eval --read-only --json github:$nixos_23_05#qemu.outPath "$@"
}
clear_cache() {
    rm -rf /tmp/nix-cache
    mkdir /tmp/nix-cache
}
export -f nix_eval clear_cache

hyperfine --warmup 1 --prepare clear_cache \
          'nix_eval baseline' \
          'nix_eval zipball' \
          'nix_eval git-accessor'

# Benchmark 1: nix_eval baseline
#   Time (mean ± σ):     14.380 s ±  1.837 s    [User: 2.233 s, System: 5.512 s]
#   Range (min … max):   11.677 s … 17.357 s    10 runs
#
# Benchmark 2: nix_eval zipball
#   Time (mean ± σ):      6.146 s ±  1.951 s    [User: 2.104 s, System: 0.575 s]
#   Range (min … max):    4.254 s …  8.989 s    10 runs
#
# Benchmark 3: nix_eval git-accessor
#   Time (mean ± σ):     13.187 s ±  0.280 s    [User: 5.543 s, System: 7.273 s]
#   Range (min … max):   12.817 s … 13.768 s    10 runs
#
# Summary
#   'nix_eval zipball' ran
#     2.15 ± 0.68 times faster than 'nix_eval git-accessor'
#     2.34 ± 0.80 times faster than 'nix_eval baseline'

Results

Github input accessor	Timings	Relative Timings
zipball	6.146 s ± 1.951 s [User: 2.104 s, System: 0.575 s]	1
git-accessor	13.187 s ± 0.280 s [User: 5.543 s, System: 7.273 s]	2.15
baseline	14.380 s ± 1.837 s [User: 2.233 s, System: 5.512 s]	2.34

The download speed for the Github tarballs/zipballs was limited by Github to about 60/70 megabits per second, staying far below the internet connection capacity of the benchmark system.

The zipball approach really shines in this benchmark. The download part is a bit slower than baseline (due to the larger archive size), but there's no extraction phase.
This also reduces the write workload for zipball, which is beneficial for SSDs.

git-accessor gets around 17% (~ 2 sec) faster in this benchmark when the libgit cache already contains a not too distant nixpkgs revision, because there are fewer items to add to the cache. But even then, git-acessor is clearly slower than zipball.

When eval'ing a large NixOS system, zipball is only a few percent slower than baseline. Overall, zipball might be the best approach.

[edolstra]

This should be improved a lot by #10013.

[thufschmitt]

Triaged during the maintainers meeting.

This may be fixed by #10013, but unclear to what extent.

@erikarvstedt do you think you can run your benchmarks again on top of current master to see whether there's still a regression?