Blake hashing tracking issue

[Ericson2314]

The Blake family of hash algorithms has a number of advantages for random access because data is ingested in a tree rather than linear stream. Tvix is using it for this purpose.

In order to support such experiments and potential compatibility with Tvix's experiments in the future, I think we should the same Blake hashing algorithm that they do.

CC @edef1c

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-12-04-nix-team-meeting-minutes-200/57005/1

[silvanshade]

I have some thoughts on this based on experience implementing support for BLAKE3 in my personal Nix fork.

I submitted an initial version based on the single-threaded C BLAKE3 implementation in the PR #10600

I opted to close the PR shortly after since I had another version that used the Rust-based multi-threaded Rayon implementation which was much faster under the expected circumstances.

Ultimately, I never submitted a second PR using the Rust version for a few reasons.

The Rust version required a fairly substantial amount of additional infrastructure code in order to be able to both define bidirectional C++ <-> Rust bindings (in a clean, extensible way) and also to be able to support building and linking the Rust code along with the rest of the C++ build process.

There was also some refactoring I had to do with how the hashing API was defined on the C++ side since, in order to take advantage of the full performance of the Rust implementation, I needed to allow the Rust implementation to handle IO. The default behavior of the Nix implementation was to use chunks that were too small to take advantage of parallelism and also did not provide support for memory-mapped IO which is the fastest scenario for the Rust BLAKE3 implementation.

I solved these issues and had a working version but at the time I was unsure about whether it was a good idea to submit something potentially that substantial just in order to have a more performant BLAKE3 implementation.

I also felt that there might be some dissatisfaction with the approach given how the bindings were defined since there wasn't much support for automation (e.g., around bindings for the hash types in either direction). bindgen is generally not appropriate for this sort of thing since it often generates incorrect and unsafe code for C++ and even using cxx still requires a lot of boilerplate.

To give a bit more context, I've been working on high quality LLVM and MLIR bindings for Rust, using the libtooling and unstable C++ APIs, via a new approach that leverages C++ type traits, concepts, and general metaprogramming facilities, and wanted to try to apply that approach here to allow for more extensibility and less boilerplate. This new approach allows for being able to derive Rust traits for the various Nix C++ types automatically, for example.

But it's been awhile since I've worked on adapting that for the Nix Rust BLAKE3 branch, so maybe it would be better in the meantime to revive the earlier Rust version I did have working for Nix, and create a draft PR based on that, at least to serve as a reference point for discussion about some of the implementation issues that arise.

Then maybe later on I can elaborate more on the ideas I had to make things nicer for broader interoperability between C++ and Rust in the code base.

The code I had is based on the Nix repo from some months ago, before the Meson changes were starting to be introduced, so I'm not sure what it will be like to bring that back up to date. But I'll try to open a draft PR if I can get things building again.

[Ericson2314]

@silvanshade I sort of feel it might be best just to land the C thing after all?

I have nothing against Rust and better bindgen --- I like those things a lot! --- but I would rather land some simple-stupid-slow BLAKE3 now, and replace it with something better later :). This is especially as since we just switched to Meson, we are perhaps a bit "fatigued" on process workflow changes in the short term (emphasis on short term!), if that makes any sense.

I don't think there will be a perfect-vs-good situation here --- if Blake hashing becomes popular, then I would expect more desire for better perf, more fun things from Tvix, etc., not any complacency that the existing C stuff is "good enough".

(Separately, I would encourage you to take your bindgetn expertise to the Rust bindings in https://github.com/nixops4/nixops4. Completely independently of BLAKE, if we can do Rust things better, we can stop C++-ing sooner :))

[silvanshade]

@Ericson2314 Okay, that sounds fine to me.

I can re-open and update the linked PR for the C implementation.

[Ericson2314]

@silvanshade that would be fantastic! Please do! :)