Nar uses lazy IO instead of streaming

[domenkozar]

Feed it a store path with lots of files and you end up with:

/nix/store/3m1hc2mrz6zhqng088s4fw0rzfb1g6qy-perl-5.28.1: getDirectoryContents:openDirStream: resource exhausted (Too many open files)

[roberth]

The culprit.

One may also consider unix's bytestring directory functions, to match Nix more closely. For performance, it's best to getFileStatus once per path.

[shlevy]

@imalsogreg Are you able to take a look?

[imalsogreg]

Yeah - thanks for the analysis on this. I'll try to reproduce with a test and then fix with some combination of streaming and unix. Will update the ticket when I have a WIP branch.

[imalsogreg]

The vigorous-io branch includes a test that can hit the resource exhausted (Too many open files) error.

Right now I have the test set to create just 100 files and assert that 50 file descriptors are open after nar serialization. If I raise the number to 10000, we see an exception. Maybe the test should use 10000 and watch for the exception.

Anyway, now to code up the solution :)

[domenkozar]

Relevant: #52

[domenkozar]

Hey @imalsogreg did you make some progress on this one?

[imalsogreg]

@domenkozar Nope, attention's been elsewhere, but thanks for the bump.

[domenkozar]

I'm happy to sponsor $150 for someone fixing this issue.

[imalsogreg]

@domenkozar I've been focusing on this the past couple days, on the vigorous-io branch.

[sorki]

Closing since this landed some time ago, it could use some real world testing! 😄