WIP: Rust: Add support to install/update crates.io wraps #11727
Conversation
It was moved a while ago in msubprojects and is now dead code.
It share code with wrap update command and have more consistent output and CLI.
This reflects better that they could be used for other sources that wrapdb.
This looks for a Rust project in crates.io and generate a wrap file for it. The subproject name and the dependency it provides have "-rs" suffix to distinguish from WrapDB wraps.
You probably want to include the API version in the dependency name, i.e.
This should probably get some semver handling by default. Just shortly commenting now, will look closer later :) |
🤔 I wonder if we should do this for C/C++ too. |
rustc will generate versioned symbols in the ABI by default? How do we know that API version from crates.io? Is it assumed to be the first 2 numbers in e.g.
We already do it manually, for example |
Even better (or worse?): it adds a hash over the API and its dependencies at the end of each symbol. So you have e.g. aka
If the first component is > 0 then you take only the first component (i.e. 1.0.0 and 1.99999.123 are compatible but 1.0.0 and 2.0.0 are not), otherwise you take the second component (i.e. 0.20.0 and 0.20.9999 are compatible, but 0.20.0 and 0.21.0 are not). |
Yeah no I'm being sarcastic. Diamond dependency hell isn't supposed to be something you solve by building both copies and versioning the symbols. Projects like glib are very much the exception, and a sort of useless exception since effectively only 2.0 actually exists.
Still better than hasklol, which hashes the command line passed to the configure command -- which means that the libfoo-$hash.so filename output changes "ABI" when you toggle tests on or off. |
In Rust this is relatively common because it's quite painless. You can easily end up with multiple versions of the same dependency in your build because everybody is moving to a newer version at their own pace. As long as the dependency is not exposed via public API and you mix the same types of both versions everything's going to work just fine. And if you do the compiler will reject your code even before linking and tell you that you're mixing two different versions. The alternative would be for the whole ecosystem to update to newer versions in lockstep, which simply does not scale. See also why we're still stuck with GLib 2. |
|
I also forgot to add that this really needs to be handled somehow or otherwise building a bigger Rust project with lots of dependencies is going to be a bit of luck. For example, right now it's going to be hard to not have |
And if both the major and the minor version are zero, then you compare the patch version. 0.0.6 and 0.0.8 are considered incompatible. The official rule from the docs is:
|
|
For the record I'm officially NACKing anything that involves teaching There's a couple reasons that I don't like the idea, but there is absolutely no way at all that I'm even slightly comfortable implementing algorithms for determining whether or not two dependencies are "compatible enough". This works for glib and similar C/C++ libraries because those projects have gone out of their way to provide robust buildsystem support for avoiding clashes at multiple levels -- shared libraries, command-line tools, man pages, data files, etc. -- and then explicitly marked the dependency name themselves, and made those dependency names non-conflicting by adding manual versioning to the names. It is not meson's job to know how to make this judgment call. Stop trying so darned hard to convince me that it's not meson's job to know what the rust programming language is1. 😂 Footnotes
|
So basically how it works in C/C++, gotcha.
So kind of like how it works in C++. Also in C but opt-in via common technology like symbol versioning scripts.
I doubt that has the slightest relationship to why we're still "stuck with" glib-2.0. It was, after all, carefully designed to do what rust just magically does for you using heuristics. Even if that weren't the case, glib still makes it possible to do exactly what C/C++ libraries have done for a long time, and make your code build on both versions, which lets you test that it works on both versions, then let system integrators build a single monolithic stack against whichever version of glib they wish to use. Rust makes it easy to not support this. 😂 |
The "algorithm" here is simple semver, extended to be stricter for <1.0 versions (semver allows everything there, Rust has the same rules as for >=1.0 shifted by one component). That's enforced by the ecosystem, there's tooling for checking compatibility (for maintainers to make sure their releases don't break the rules), and it's part of dependency resolution. A tool like proposed in this PR will need to implement that to select the correct versions / release series of a dependency, or else it simply won't work. You have some level of consistency and know what to expect without learning each project's numbering scheme, just like with e.g. npm. Which is also why it actually works as part of tooling and doesn't require workarounds like including the API version in the name.
Like you said multiple times yourself, "So basically how it works in C/C++". Just that it's built-in instead of constructed on top and handled slightly differently by different projects.
Except that it was not (it's parallel installable though). For GLib/GObject at the API level (it's going to be part of your public API so you can't mix two versions in the same build target), for anything on top of GObject also at the ABI level even if they wanted to (GObject type names for example, see e.g. libsoup). But we're digressing.
I'm not sure what magic you're talking about. There's no magic but very simple rules consistently used by the whole ecosystem. If there was magic (aka complex rules and heuristics) then this wouldn't actually work in practice. I have the feeling we're talking about completely different things, and you're talking about something unrelated to the problem at hand in this PR but I'm not sure what exactly. |
|
As you pointed out above, rust does not make this work, it just promises to make it a compiler error if you mess up. Same thing C++ promises. I don't know that this can actually be guaranteed 100% though, probably only 99.9%, because one dependency "A" could be directly depending on "C" 1.x and making decisions based on it, and passing those decisions as booleans upward to projfoo, which then calls into dependency "B" that utilizes those decisions against "C" 2.x. Not every compatibility break is a type violation. Dirtying the tree upwards is actually just saying "resolve a single consistent version everywhere". I mislike the idea of meson creating new autogenerated wraps that perform version suffixing and relax the usual rules which make this sort of thing a configure time error rather than a build time or runtime error, and I certainly mislike the idea of changing the subproject dependency resolution algorithm to make this work the way rust does under the hood. It's just asking for trouble. If project maintainers wish to manually rename their dependencies, they can do that for C also, and this will work with rust exactly as you wish. |
That's not the idea here. What I think would make sense is to generate The only potential issue is with installed files, if the crate is not made to also add version into installed library/executable names. But that's something we can do automatically in cargo subprojects. ATM it does not install any file anyway AFAIK. |
|
That's irrelevant for this PR though, just loosely related. If there are multiple versions of the same crate appearing somewhere (and that will happen), then it's your job to make sure things fit together (i.e. not multiple versions of the same crate being used interchangeably in API as that will give a compiler error). That's not meson's job, and can't be meson's job, but whoever puts together the set of subprojects. However what has to happen in this PR is
|
|
Note that fetching dependencies recursively is a whole other story that I discussed with @thiblahute in PM last week. I'm not sure exactly how that will work. The problem is making Deps could come from Cargo.toml or Cargo.lock as git URL, etc. My idea currently is to only write .wrap files for direct dependencies using this PR. Once Meson configure a cargo subproject it generates PackageDefinition objects (the class that represents a parsed .wrap file) based on info from Cargo.lock. See that as in-memory wraps that have no .wrap file written. This is similar to normal subprojects that have git submodules in their subprojects/ without corresponding wrap file. |
|
Yeah let's worry about that separately, I guess. Handling dependencies recursively could be a future step.
|
|
So, I'm not really familiar with meson and thus chances are that this is naive, but why not just let I can understand wanting to handle |
|
I tried this out, and I have a concern about the automatic suffixing with "-rs": When I have a missing dependency, it'll complain about whatever-rs being missing. If I'm not paying attention, I'll try to It's also a bit unintuitive. I understand that sometimes you'd have both C and Rust gstreamer etc. libraries, but that's probably the exception, right? Wouldn't it make more sense to have a way to rename dependencies in case of conflict, rather than renaming all Rust dependencies by default? |
|
I'm not sure what you actually mean by "rename dependencies in case of conflict", but keep in mind that the major conflict is probably not between C and rust implementations of the same project, rather, C and rust libraries that both try to do some common thing, and weren't previously concerned about using the same project name to do it with. |
|
The problem is Cargo is its own namespace for project names. Unlike the rest of the world that needs to make sure to have a unique e.g. pkg-config name, crates only have to be unique within crates.io. It does not seems to be the exception, pretty much every C library wrapped in Rust have a potential name conflict. The problem becomes more important with #11856 where I also use the Note that I use
We could easily allow both. To be fair, I'm not completely sure we want |
This adds
meson wrap install --crates-io gstreamerthat generatessubprojects/gstreamer-rs.wrap:meson wrap updateis capable of detecting that this wrap is from crates.io and update to latest version (by parsingsource_url).A few notes:
-rssuffix to the wrap name (and thus subproject name). This is because we could have bothgstreamerandgstreamer-rssubprojects, the former is the C project and the latter is its Rust binding.meson.override_dependency('gstreamer-rs', dep), with that name (i.e. crate name with-rssuffix). Again to distinguish fromgstreamerpkg-config dependency.meson.buildfile upstream, similar to those @sdroege wrote manually. But in practice it will require @dcbaker's Rust module that can configure a subproject from Cargo.toml.meson wrap updateshould make some sanity checks to ensure all dependency version requirements are met.