Add BLAKE3 hashing algorithm

[silvanshade]

This uses the single-threaded C-based routines from libblake3.

This is not optimal performance-wise but should be a good starting point for nix compatibility with BLAKE3 hashing until a more performant implementation based on the multi-threaded BLAKE3 routines (written in Rust) can be developed.

@Ericson2314

Motivation

See below.

Context

#10600 #11999

EDIT: Before merging this please see the companion PR with BLAKE3 via Rust interop: #12416

[roberth]

Nice!
Have you cross-validated this with tvix's use of BLAKE3 in their store implementation?

[silvanshade]

Have you cross-validated this with tvix's use of BLAKE3 in their store implementation?

No, but I'm also not sure if that is a meaningful thing we can do in this context. Maybe you can elaborate if you have something specific in mind?

I don't know much about tvix but from scanning the documentation and source code it appears that their use of BLAKE3 is primarily for their custom data model and protocols around interacting with the content addressable store, which as far as I know is not directly comparable to the equivalent in C++ nix.

It would theoretically be possible to re-implement what they have done for the castore here but it would be a lot more work, beyond the scope of this PR, and certainly beyond my current knowledge of nix internals.

What this PR does is just add BLAKE3 as another hashing algorithm, which could presumably be used for anything nix does internally related to hashing, but at the moment is only really exposed for SRI hashes and via the CLI interface for nix hash.

Surprisingly, tvix doesn't seem to support BLAKE3 for SRI hashes though: https://github.com/tvlfyi/tvix/blob/f8325a64845500916f1e40f5d04466d4c7d1bef6/eval/src/builtins/hash.rs#L19-L29

[Ericson2314]

Looks good to me!

[Ericson2314]

I asked in #tvix-dev

[Kranzes]

Surprisingly, tvix doesn't seem to support BLAKE3 for SRI hashes though: https://github.com/tvlfyi/tvix/blob/f8325a64845500916f1e40f5d04466d4c7d1bef6/eval/src/builtins/hash.rs#L19-L29

Because we're trying to be compatible with with Nix.

[edolstra]

Team meeting notes:

• Decided to merge this as an experimental feature. We can drop the experimental feature status if there is evidence of actual use. @silvanshade Can you add the experimental feature?

[silvanshade]

• @silvanshade Can you add the experimental feature?

@edolstra Okay, I can add the experimental feature. What should I use as the tracking URL?

[silvanshade]

I made some additional changes:

• Added the blake3-hashes experimental feature
• Added an optional ExperimentalFeatureSettings argument to Hash and hashString
• Switched from TEST to TEST_F to allow enabling blake3-hashes feature in tests
• Added an example BLAKE3 hash digest from https://datatracker.ietf.org/doc/draft-aumasson-blake3/
• Added blake3 to the documentation (blake3 will already show in error messages so might as well put it in docs)

[Ericson2314]

We can add the milestone after, thanks!

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-02-12-nix-team-meeting-minutes-212-5/60216/1