rust: cargo subprojects: Add some kind of support for non-default crate registries, local source code

[sdroege]

Fedora (and AFAIU Debian, @werdahias can confirm / correct) are currently building Rust code by having all dependencies locally available on the system, and telling cargo about that. It will be necessary to add a similar mechanism for meson so that cargo subprojects can be built without network access and without having to patch the code more than necessary.

For cargo, Fedora currently sets the following .cargo/config.toml in each project:

[source.local-registry]
directory = "/usr/share/cargo/registry"

[source.crates-io]
registry = "https://crates.io/"
replace-with = "local-registry"

This makes it take the crates from the local file system instead of doing network access to crates.io.

I'm not sure how to expose something similar in a meaningful way with meson, but the most suitable way would probably be via some kind of -D configuration?

Also, generally for when adding cargo subprojects to the project it would probably be good to be able to point at non-default crates registries. While almost everything is on crates.io, it seems good to stay generic here.

CC @decathorpe @werdahias @xclaesse

[sdroege]

FWIW, finding a solution here would be necessary for packaging mesa once there's Rust code using external dependencies as well as GStreamer (its Rust code currently does not use external dependencies because of this specific issue, but that's not sustainable).

[werdahias]

@slomo for projects using meson and rust mostly we use this in debian:
This calls a cargo wrapper script and sets some variables linking the "local" registry of crates (/usr/share/cargo/registry) into $builddir/debian/cargo_home and passing that to meson. It's not very seamless or elegant tbh. Here is an example of what this looks like for a real package.

[sdroege]

That's for projects calling cargo from meson, which should work more or less as with plain cargo. This here is about cargo subprojects in meson, which are not calling into cargo at all.

[eli-schwartz]

currently building Rust code by having all dependencies locally available on the system, and telling cargo about that. It will be necessary to add a similar mechanism for meson so that cargo subprojects can be built without network access and without having to patch the code more than necessary.

Since this is handled via meson subprojects, my assumption was this would work out of the box by creating wrap files.

This works perfectly fine without network and without patching code, because meson is always very careful to use the meson subprojects cache if it is possible to do so; while it's generally a bad idea to use subproject fallback at all for distro packaging of C/C++ projects as usually a prebuilt dependency should be preferred, the general solution is still possible -- download the additional tarballs referenced in wrap files alongside the main project tarball, unpack the main project tarball, and copy/symlink the additional tarballs into subprojects/packagecache/ of the main project tarball -- or extract them to subprojects/ and make sure you manually handle any patch overlay or diff_files.

meson dist will also optionally distribute subprojects in the main project tarball for you. I would actually recommend this approach for Mesa and GStreamer because every Linux distro wants offline builds, but only two of them are recreating the cargo registry protocol.

[xclaesse]

An easy way would be to dist subprojects with mesa/gstreamer, with meson dist --include-subprojects. But I guess that won't be acceptable?

IMHO, we should have a generic way to tell meson to search for its subprojects elsewhere. I already have a WIP branch to have MESON_HOME=~/.meson to share downloaded tarballs there (replace packagecahe dir). I was experimenting with cloning git repo there as well and add a worktree in subprojects/ automatically.

In our case we could have MESON_HOME=/usr/share/cargo/registry, it would have to support having an already extracted tree there instead of tarball. I think it would have to copy the source into subprojects/ because it's likely we'll need to patch cargo projects to add meson/meson.build file where we port manually build.rs logic.

[sdroege]

meson dist will also optionally distribute subprojects in the main project tarball for you. I would actually recommend this approach for Mesa and GStreamer because every Linux distro wants offline builds, but only two of them are recreating the cargo registry protocol.

Shipping copies of everything is not really a scalable approach, and specifically Fedora/Debian would want to have their versions of the crates being used instead of whatever happened to be shipped with the tarballs.

What currently is already possible it to patch things and copy over the source from the distro packages (from /usr/share/cargo/registry) but that's a very manual approach.

In our case we could have MESON_HOME=/usr/share/cargo/registry

I was thinking more of something specific to Rust crates as the layout in /usr/share/cargo/registry (and generally crates registries) is something more specific than just a directory with random subproject tarballs. (Also for non-Rust subprojects, Linux distros probably want to not use the subprojects but use them as external dependencies?)

[xclaesse]

(Also for non-Rust subprojects, Linux distros probably want to not use the subprojects but use them as external dependencies?

Distro typically build with --wrap-mode=nodownload, which means fallback is fine as long as source is available locally. e.g. glib dist with gvdb subproject.

[werdahias]

@sdroege sorry, got confused there for a sec. Adding support to meson for local / distro repository for subprojects seems sensible.

[xclaesse]

I was thinking more of something specific to Rust crates as the layout in /usr/share/cargo/registry (and generally crates registries) is something more specific than just a directory with random subproject tarballs.

$ ls /usr/share/cargo/registry/
proc-macro2-1.0.47/  syn-1.0.107/         
quote-1.0.21/        unicode-ident-1.0.0/ 

I don't see anything specific to Rust/Cargo here, that's exactly the layout you would expect to have in subprojects/ for any wraps. It's like pointing packagecache there, but instead of only searching for the pre-downloaded tarball, also search for pre-extracted directory. That's is a bit what I want to allow to share git clones for example.

[eli-schwartz]

Shipping copies of everything is not really a scalable approach, and specifically Fedora/Debian would want to have their versions of the crates being used instead of whatever happened to be shipped with the tarballs.

So what exactly is your suggestion for the dozen-plus distros that aren't Debian or Fedora and do, in fact, want to ship copies of everything? Tell them that it's not a scalable approach and they better get as many maintainers as Debian, or get funding from a big enterprise distro, so that they can repackage all of cargo by sticking giant wads of unmodified source code into binary packages and call that a classic distro dependency "because it comes from a package"?

Unfortunately, there's no good answer here because Rust The Programming Language does not in fact have a good usability story for distributing partially precompiled dependencies, metadata, last-mile generics resolution, and interface exports, the way other languages do. Very few third-party entities are willing to invest time in the topic because it feels like a waste of time with no actual gain to packaging source code and rust devs don't feel like the general issue is an actual problem to solve. So there's nothing for meson to use there either.

The one thing that does exist is reimplementing crates.io using a file:/// access protocol, which is... tremendous busywork. It doesn't solve the problem, and every distributor would have to do it independently just to "solve" the networked builds issue.

Why is it important that a project which is statically compiling and building all of its dependencies, uses "the system copy of the source code" by anyways patching the project sources to use it? IMHO since Debian absolutely loves busywork (both doing it themselves and making it for others) they can do a tiny bit more.

Also for non-Rust subprojects, Linux distros probably want to not use the subprojects but use them as external dependencies?

Yes, but those work today...

[eli-schwartz]

@xclaesse the idea of a central location for subprojects sounds interesting, but I don't think it's practical to both do that and apply overlays/patches. If nothing else, we use the existence of a subproject directory as an indicator that we have fully initialized it.

However, perhaps it would work if we assume that Debian actually does, indeed, apply patches to crates they have packaged into binary source code. They could apply meson patches to any crates whose reverse dependencies need them for meson. :)

[nirbheek]

I think we're overthinking things here, and trying to come up with abstractions that don't need to exist (because there's only one user for it).

I think Meson should try to be as much of a drop-in replacement for Cargo as reasonable and feasible. Our aim should be to enable people to have their projects work with both Cargo and Meson without doing too much extra work, wherever we can enable it without bending over backwards. This applies to maintainers of projects, consumers of projects, and integrators of projects (distros).

So, we already read Cargo.toml for the build definition, why not also read it as the equivalent of wrap-redirect (with built-in support for crates.io) and read .cargo/config.toml as user configuration for redirecting from crates.io to elsewhere? That will also help orgs that maintain subset-mirrors of crates.io (not as hard as you'd think).

It's just simple configuration files, not a turing-complete mess like cmake. If we warn users about ignored sections/fields in .cargo/config.toml that would be enough.

It's absolutely in our best interests to ease the transition of people from using Cargo to using Meson.

[sdroege]

Very few third-party entities are willing to invest time in the topic because it feels like a waste of time with no actual gain to packaging source code and rust devs don't feel like the general issue is an actual problem to solve. So there's nothing for meson to use there either.

Except that cargo actually has a solution for this, which people would lose if they use meson instead. Like @nirbheek said,

It's absolutely in our best interests to ease the transition of people from using Cargo to using Meson.

And it's not only an issue for some Linux distros but also by other organizations that want control over their software supply chain (and usually they don't end up with strange contraptions like Fedora/Debian that require packaging every crate individually).

---

For directly using .cargo/config.toml, that seems like an OK solution to me. Note that the config can be per project but also cargo-global (e.g. ~/.cargo/config.toml).

[xclaesse]

@nirbheek It's on my TODO list to make Cargo.toml (or Cargo.lock?) some kind of wrap provider. Basically we can just create PackageDefinition objects from Cargo the same way we do by reading subprojects/*.wrap. But I think that's unrelated to this issue which is about how to get the source code.

but I don't think it's practical to both do that and apply overlays/patches.

My idea right now is to basically cp -r /usr/share/cargo/registry/proc-macro2-1.0.47 subprojects/ then apply patch/overlay there just like any normal wraps.

[nirbheek]

@nirbheek It's on my TODO list to make Cargo.toml (or Cargo.lock?) some kind of wrap provider. Basically we can just create PackageDefinition objects from Cargo the same way we do by reading subprojects/*.wrap. But I think that's unrelated to this issue which is about how to get the source code.

As part of creating those PackageDefinition objects you'd also implicitly look at crates.io for getting the source code, right? Like with a section like this in Cargo.toml:

[dependencies]
log = "0.4"
env_logger = "0.8"
futures = "0.3.10"
async-std = { version = "1.0", features = ["unstable"] }
glib = "0.10"
glib-sys = "0.10"
gst = { version = "0.16", package = "gstreamer", features = ["v1_18"] }
once_cell = "1"
serde_json = "1"

All these are implicitly fetched from crates.io, and with .cargo/config.toml the source is redirected to a local directory. Isn't that exactly what this issue is about?