Spin out libfetchers into stand-alone project

[roberth]

Is your feature request related to a problem? Please describe.

Reproducible and lazy fetching from multiple kinds of sources is a well-defined problem.
It has been questioned whether this should be a responsibility of the Nix project. Notably tvix has so far refused to implement fetchTree (iiuc).
Multiple projects could benefit from this functionality.
Having a single, reusable implementation of this functionality in the ecosystem helps with its goal of producing reproducible outputs.
A division of labor, which is feasible here, improves the efficiency of both teams.

Describe the solution you'd like

• Review the libfetchers interface
• Make libfetchers tests independent of Nix
• Review libutil dependency
• Move libfetchers into its own repository.

Describe alternatives you've considered

• Specialize without refactoring, and keep in the same repo. This makes libfetchers unattractive to other implementers.
• Use git subtrees for everything, and have a tool to manage the subtrees.

Additional context

Discussed at #ZurichZHF with @fricklerhandwerk

Priorities

Add 👍 to issues you find important.

[Ericson2314]

I would say

1. Meson build
2. Use Meson subproject to build each library in own derivation
3. Move fetchTree primop and libfetchers -> libexpr to additional library (primops can be registered downstream in plugin fashion)

Step 4 would be making that additional library work as plugin, but we'll have to untangle some CLI stuff.

Future work could be be ABI stability and actually moving things out of tree, but I don't feel so urgent about that.

I do think having the core SourceAccessor stuff remaining in libutil and be used by libexpr is good, however.

[fricklerhandwerk]

Yes, whether something is in the tree doesn't matter, it's all about strong boundaries in interfaces and ownership. Namespaces merely help with delineating responsibilities.

Side note: shouldn't libutil become a library for file system handling then?

[edolstra]

The use case for this is unclear to me. What other project would use libfetchers? It's extremely Nix-specific. Meanwhile it would make hacking on Nix much more painful.

[roberth]

I believe it is feasible once we've narrowed down the interfaces. I wouldn't exclude the possibility that a truly different project wants to use it, although I am not aware of an example. I could see Lix or Tvix use it though. I haven't been in touch with them about this yet, but I believe the concept of reproducible fetching has a better chance of success when there's a serious project that has this as its single purpose. Specifically, it allows for more collaboration, and it is the cheapest way to have consensus about what any fetchable source should evaluate to. The alternative, competing implementations that converge on a standard, takes much more effort, and neither team seems particularly excited to work on fetchers, from what I can tell.
Without a suitable standard implementation, the only remaining choices for them are to either: give up on the concept of lazy reproducible fetching, or work on an alternate implementation that diverges (such as stabilizing the buggy 2.18 implementation in an incompatible way).