empty derivation and can't repair the various wrong store paths, thus blocking nixos-rebuild

[teto]

Describe the bug

I find myself in a situation where I can't rebuild my system and I am not sure how to repair the store without manually deleting lots of derivations.

Upon nixos-rebuild, I've got an error I've never seen before:
error: error parsing derivation '/nix/store/8ihsdsn00n2p9hy9hz2jmw5z5wfl3w9r-nss-3.102.drv': error: expected string 'D'.
When I look at /nix/store/8ihsdsn00n2p9hy9hz2jmw5z5wfl3w9r-nss-3.102.drv, it's empty :/

sudo nix-store --verify --check-contents shows around ~80 failures in the store (wrong hash, which might be because of unexpected shutdowns?!. The command finishes
with the weird : warning: not all store errors were fixed.. Since it was run without --repair, i dont think it tried to fix any error so this specific message is weird: either no message or warning: no repair was done: rerun with --repair.

I then tried to actually repair the paths:

error: cannot repair path '/nix/store/aigwdcmwand90dasgsh3837p5n5zkvac-swaybg-1.2.1.drv'
path '/nix/store/apc6ihyrk677wb5q6dw80qbjz1nhvd8x-sway.conf.drv' was modified! expected hash 'sha256:15830m1af8hv8mz1y403ajh9lqibb4y3p46py9b5icviv51ghalh', got 'sha256:0ip26j2h11n1kgkz36rl4akv694yz65hr72q4kv4b3lxcbi65b3p'
error: cannot repair path '/nix/store/apc6ihyrk677wb5q6dw80qbjz1nhvd8x-sway.conf.drv'
path '/nix/store/b0llvi2bw2hhlfl1nimmbz938prf2zwg-qtlanguageserver-6.7.2.drv' was modified! expected hash 'sha256:1wy52gqzkya5qwcl6jrlgvzas6gs6ay7bg73dxq088i7368yk72f', got 'sha256:0ip26j2h11n1kgkz36rl4akv694yz65hr72q4kv4b3lxcbi65b3p'
error: cannot repair path '/nix/store/b0llvi2bw2hhlfl1nimmbz938prf2zwg-qtlanguageserver-6.7.2.drv'

It's not clear why it can't repair the path ?
( as a sidenote I also tried:

nix store repair --all 
error: operation 'repairPath' is not supported by store 'daemon'

)

so I thought I would nee

sudo nix-store --ignore-liveness --delete /nix/store/2zjwchn4vwcpbqxfzi24fmzzdcgibs9b-source.drv 
finding garbage collector roots...
0 store paths deleted, 0.00 MiB freed
error: Cannot delete path '/nix/store/2zjwchn4vwcpbqxfzi24fmzzdcgibs9b-source.drv' since it is still alive. To find out why, use: nix-store --query --roots and nix-store --query --referrers

I am not sure how to proceed to unblock the repair. Maybe delete all the referrers manually ? Isn't it possible to force delete a .drv store path ? --ignore-liveness could become --force instead for a more standard flag.

nix-env --version output

nix --version         
nix (Nix) 2.23.1

Additional context

My laptop oftenu shut down unexpected recently for various reasons, which could explain the wrong store paths.

Priorities

Add 👍 to issues you find important.

[teto]

garbage-collecting everything unblocked me but this doesn't seem like a good workaround. I'll leave this open in case the maintainers want to requalify the UX problems (more actionable messages: why can't nix repair the pat ? when --check ing, mention how to --repair etc) I mentioned but feel free to close otherwise.

[roberth]

.drv files are often not pushed to a cache, in which case the only way to restore them is to perform instantiation again, ie evaluate.
Currently these files are not checked; the instantiation process assumes that "valid" (ie extant) store paths are not corrupted.
That's a choice that was made for performance, but with an extra bit in the database, we could get both performance and automatic recovery.

TODO:

• Improve error message and/or improve
  • Suggest sudo in

    nix store repair --all 
    error: operation 'repairPath' is not supported by store 'daemon'`
    

  • cannot repair path -> say it wasn't substitutable
• Remember corrupted path state in store db
  • Check asynchronously that build dependencies aren't corrupted #8656
• Make instantiation recover corrupted paths

[Elfahor]

I'm facing this, but I haven't been able to unblock myself by calling the GC

[Scrumplex]

I am also facing this. For a few weeks, one of my machines keeps hard resetting irregularly, and it seems like it did so during a larger evaluation/build :/

I was able to delete affected gcroots, garbage collect and run a repair afterwards.

[n8henrie]

I've run into this a few times now. Commenting in part to help me find this thread again in the future since error: expected string 'D', got '' is somewhat hard to search for.

I'm getting the error upon trying to nixos-rebuild build on /nix/store/4ppns4azi7nnk9s31s0v9rbzbcj65lh5-unit-systemd-random-seed.service.drv

The file is empty:

$ file /nix/store/4ppns4azi7nnk9s31s0v9rbzbcj65lh5-unit-systemd-random-seed.service.drv
/nix/store/4ppns4azi7nnk9s31s0v9rbzbcj65lh5-unit-systemd-random-seed.service.drv: empty

Deleting fails, with both new and old style commands:

$ sudo nix store delete --ignore-liveness /nix/store/4ppns4azi7nnk9s31s0v9rbzbcj65lh5-unit-systemd-random-seed.service.drv
warning: The interpretation of store paths arguments ending in `.drv` recently changed. If this command is now failing try again with '/nix/store/4ppns4azi7nnk9s31s0v9rbzbcj65lh5-unit-systemd-random-seed.service.drv^*'
0 store paths deleted, 0.00 MiB freed
error: Cannot delete path '/nix/store/4ppns4azi7nnk9s31s0v9rbzbcj65lh5-unit-systemd-random-seed.service.drv' since it is still alive. To find out why, use: nix-store --query --roots
$ sudo nix-store --delete --ignore-liveness /nix/store/4ppns4azi7nnk9s31s0v9rbzbcj65lh5-unit-systemd-random-seed.service.drv
finding garbage collector roots...
0 store paths deleted, 0.00 MiB freed
error: Cannot delete path '/nix/store/4ppns4azi7nnk9s31s0v9rbzbcj65lh5-unit-systemd-random-seed.service.drv' since it is still alive. To find out why, use: nix-store --query --roots

Querying roots fails:

$ sudo nix-store --query --roots /nix/store/4ppns4azi7nnk9s31s0v9rbzbcj65lh5-unit-systemd-random-seed.service.drv
error: error parsing derivation '/nix/store/4ppns4azi7nnk9s31s0v9rbzbcj65lh5-unit-systemd-random-seed.service.drv': error: expected string 'D', got ''

Repairing also seems to fails, but on a different file -- so perhaps unrelated? And still can't query roots afterwards:

$ sudo nix store repair --all
error: cannot repair path '/nix/store/000irn9874zybq42d4dlddby4lz6kz73-easy-file-0.2.5.tar.gz.drv'
$ sudo nix-store --query --roots /nix/store/4ppns4azi7nnk9s31s0v9rbzbcj65lh5-unit-systemd-random-seed.service.drv
error: error parsing derivation '/nix/store/4ppns4azi7nnk9s31s0v9rbzbcj65lh5-unit-systemd-random-seed.service.drv': error: expected string 'D', got '�[0m'

Repairing just that file seems to work (man that warning is annoying), but doesn't seem to have completely solved the issue (another corrupt file):

$ sudo nix store repair /nix/store/4ppns4azi7nnk9s31s0v9rbzbcj65lh5-unit-systemd-random-seed.service.drv
warning: The interpretation of store paths arguments ending in `.drv` recently changed. If this command is now failing try again with '/nix/store/4ppns4azi7nnk9s31s0v9rbzbcj65lh5-unit-systemd-random-seed.service.drv^*'
$ echo $?
0
$ file /nix/store/4ppns4azi7nnk9s31s0v9rbzbcj65lh5-unit-systemd-random-seed.service.drv
/nix/store/4ppns4azi7nnk9s31s0v9rbzbcj65lh5-unit-systemd-random-seed.service.drv: ASCII text, with very long lines (3033), with no line terminators
$ sudo nix store gc
0 store paths deleted, 0.00 MiB freed
error: error parsing derivation '/nix/store/6xcl4rbl1kpjzjd5i26hm51dr36s8d7z-unit-systemd-udevd.service.drv': error: expected string 'D', got ''

Still stuck on that other file:

$ sudo nix store repair --all
error: cannot repair path '/nix/store/000irn9874zybq42d4dlddby4lz6kz73-easy-file-0.2.5.tar.gz.drv'
$ sudo nix store repair /nix/store/000irn9874zybq42d4dlddby4lz6kz73-easy-file-0.2.5.tar.gz.drv
warning: The interpretation of store paths arguments ending in `.drv` recently changed. If this command is now failing try again with '/nix/store/000irn9874zybq42d4dlddby4lz6kz73-easy-file-0.2.5.tar.gz.drv^*'
error: cannot repair path '/nix/store/000irn9874zybq42d4dlddby4lz6kz73-easy-file-0.2.5.tar.gz.drv'

[n8henrie]

this seems to have worked (many thanks to @rvl):

$ nix-store --query --referrers-closure \
    $(find /nix/store -maxdepth 1 -type f -name '*.drv' -size 0) |
    xargs sudo nix-store --delete --ignore-liveness
$ sudo nix store gc
$ sudo nix store verify --repair --all

[n8henrie]

Unfortunately that seems to have masked but not entirely fixed the issue -- I have no more empty derivations, but I have empty store paths.

For example, part of github:n8henrie/scanbutton:

$ file /nix/store/xl9rfarmrzkbplzy6zjiiia616gadn6c-scanner-20230225/bin/scanner-20230225
/nix/store/xl9rfarmrzkbplzy6zjiiia616gadn6c-scanner-20230225/bin/scanner-20230225: empty

Trying to repair does nothing:

$ sudo nix store verify --repair /nix/store/xl9rfarmrzkbplzy6zjiiia616gadn6c-scanner-20230225                                                                                             path '/nix/store/xl9rfarmrzkbplzy6zjiiia616gadn6c-scanner-20230225' was modified! expected hash 'sha256:0cf7i9gn0chsrjm4p178r2gg6qj89m5f2nd0mbchixhlhgnmscx1', got 'sha256:1ydf7qzifspzq3618fb8wacffqlvjq8x1g2cswc016rcw34rlcz1'
path '/nix/store/xl9rfarmrzkbplzy6zjiiia616gadn6c-scanner-20230225' is untrusted
$ file /nix/store/xl9rfarmrzkbplzy6zjiiia616gadn6c-scanner-20230225/bin/scanner-20230225
/nix/store/xl9rfarmrzkbplzy6zjiiia616gadn6c-scanner-20230225/bin/scanner-20230225: empty
$
$ sudo nix store repair /nix/store/xl9rfarmrzkbplzy6zjiiia616gadn6c-scanner-20230225
path '/nix/store/xl9rfarmrzkbplzy6zjiiia616gadn6c-scanner-20230225' is corrupted or missing!
$ file /nix/store/xl9rfarmrzkbplzy6zjiiia616gadn6c-scanner-20230225/bin/scanner-20230225
/nix/store/xl9rfarmrzkbplzy6zjiiia616gadn6c-scanner-20230225/bin/scanner-20230225: empty

I can't delete it, because it has referrers

$ sudo nix store delete --ignore-liveness /nix/store/xl9rfarmrzkbplzy6zjiiia616gadn6c-scanner-20230225
0 store paths deleted, 0.00 MiB freed                                                                                                                                                                                      error: Cannot delete path '/nix/store/xl9rfarmrzkbplzy6zjiiia616gadn6c-scanner-20230225' since it is still alive. To find out why, use: nix-store --query --roots and nix-store --query --referrers

When I delete it recursively, the referrers go back to my current system so everything crashes and burns (yeah, I figured this was a bad idea -- confirmed, non-bootable system, nonrecoverable with any of the old generations):

sudo nix store delete --ignore-liveness $(nix-store --query --referrers-closure /nix/store/xl9rfarmrzkbplzy6zjiiia616gadn6c-scanner-20230225)

Is there a way that I can force rebuilding of this store path?

[rvl]

Unfortunately that seems to have masked but not entirely fixed the issue -- I have no more empty derivations, but I have empty store paths.

Hi @n8henrie, there is nothing wrong with empty store paths, e.g. pkgs.emptyFile. The problem was empty .drv files, which are corrupt derivation files.

Using nix-store --delete --ignore-liveness is only OK if you're certain it won't touch the running system or any other gcroot that you care about.

Are you still getting error parsing derivation after deleting empty derivations and successfully running nix-store --verify --check-contents ?