Integrating conan with meson via a module.

[germandiagogomez]

Hello everyone,

I would like to discuss an idea about integration with Conan that came out in one of my projects.

Currently, I am using Meson together with conan for multiplatform support.

Conan has good support for Meson via https://docs.conan.io/1/reference/conanfile/tools/gnu/pkgconfigdeps.html. It also generates a toolchain file via https://docs.conan.io/2/reference/tools/meson/mesontoolchain.html#conan-tools-meson-mesontoolchain.

The problem from the point of view of my project is that in Conan they recommend a two-step process that is non-invasive for Meson:

• the first step is to generate all your dependencies with conan
• the second step is to point to those dependencies via pkg_config_path

After that, you can do something like this:

meson setup --native-file conan-meson\conan_toolchain.ini -Dpkg_config_path=conan-meson...

This is good as long as you do not have a project that generates dependencies conditionally. If the dependencies of the Meson project are conditional, you cannot do a first stand-alone conan step. It will just not work. But I want to keep dependencies to the strictly needed. For that I need the setup step to detect dependencies and add them to a conanfile.txt that is generated during meson setup --native-file myfile.ini.

This means that the toolchain file for Meson from conan cannot be generated. It should be exactly the opposite:

• meson should generate the conan profile file instead of the meson toolchain file being derived from the conan configuration.

Once the configuration is done, two things should be done:

1. generate the conan profile
2. invoke conan in the generation step with the dependencies detected

Namely, in this model, meson toolchain files + dependencies detection is the model by which the build is driven and conan is invoked to obtain those dependencies.

I would propose to figure out a module that can:

1. generate a `conanfile.txt` transparently
2. generate the profile files for conan transparently
3. invoke conan 
4. make it possible to consume the dependencies generated by conan

This way all the dependencies generation + conan would be integrated transparently.

[eli-schwartz]

I'm not exactly sure what you're asking for, here.

Do you mean that meson should configure the project, collect each dependency() invocation, and write out a conan file containing all the dependencies, then run conan install and re-configure meson with the resolved dependencies?

This fundamentally cannot work, for a few different reasons but especially because the detected version of a dependency can be read in a meson.build and used as a conditional rule for any other meson code, including "choose which other dependencies to look for".

So the dependencies need to be fully resolved immediately as soon as dependency() is processed.

[dcbaker]

There's really only two ways I could see to achieve this:

1. teach meson introspect to generate a list of dependencies it needs without a full configure, and feed that into Conan, continue as usual
2. build a low level interchange format so that Conan and Meson (presumably it would need to be generic enough that other package managers like vcpkg and build systems like CMake could also use it), and have Meson invoke Conan and have Conan respond with a description of how to build the target in question.

One of these sounds a lot harder than the other way.

[eli-schwartz]

Option 3:

Add a new dependency method, similar to pkg-config.

conan-tool libfoobar --modversion

But unlike pkg-config which responds to tell you whether it is installed and what version it has, conan-tool would just hang for a few minutes while it tries to install that one package for you, or returns a failure (== "dependency not found" to meson) when it cannot install the package.

Repeat for every package, one at a time. No batch jobs involved, which may confuse users if they don't get what's taking so long.

[dcbaker]

I'm not crazy about that plan, but I wouldn't NAK it if it solves users problems and someone wrote it.

[germandiagogomez]

@dcbaker that is why I am here. First, something needs to be figured out.

Let us say I could do

mydep = dependency('something') <--- detect through conan, do conan install with correct profile, and provide it.

and now think the meson wrapdb way (which is a dependency provider. What if that could be achieved with a conan dependency provider? Maybe with some limitatiions, I am not sure. But it would be great.

At the end, what Conan does is to trigger a build if needed (configurable) via conan install mypackage/1.2@ --build=missing|never (not sure about whether the version info should be there, that is not how it works for normal dependencies).

So probably it could be done lazily and per-package and consider Conan another package provider?

I like wraps but they have one thing where it cannot compete with Conan or Vcpkg: the recipes went through a lot of testing and it is nice and all to have wraps, but it is not realistic to try to catch up by authoring meson.build files for (as of Conan 2.x) 1500 packages and growing.

So I would like an as-smooth-as-possible integration with things that already work.

In my project, concretely, I migrated from meson wraps to conan as much as I could because in my experience the multiplatform support is more reliable (I am currently compiling for windows, linux and mac, dropped emscripten recently due to a blocker that is not in the build system itself but maybe will come back to it some day).

Things that need to be done would be:

- align the native file (and cross file) configuration with conan profiles. Namely, generate internally profiles
- per-package detection of some kind. Probably lazy invocation of conan install per-package generating pkg-config files. 
- consume dependency as usual.

And use it as wraps are used as much as it is feasible.

[germandiagogomez]

A proposal I would like to make that would also be useful for vcpkg: integrating conan or any dependency provider (but out of scope in my case, I would only work on integrating conan) in expressions such as

mydep = dependency('somelibrary')

This already works for wraps and fallbacks. It should not be important what happens under the hood as long as:

1. the providers can be controlled
2. the dependency can be detected lazily, not ahead of time. This eliminates the need for a conanfile.txt or similar? Not sure, but it could be the case.

The way I imagine, because I do not know it is even possible, is to make it work transparently and be able to have dependency providers as transparent as possible but selectable, as of now.

So this would amount to use a conan provider. In the conan case the profile files would be generated from the meson toolchain files directly.

What do you think as a first idea?

[germandiagogomez]

Following the discussion on what I said above. Just wild ideas so far. I would be grateful to get feedback to rely on a design that is potentially useful for plugging other providers and with some consensus in case I start an implementation. I would also like that it is potentially useful to plug in other providers.

How about extending wraps?

[wrap-conan]
package = capnproto/1.0.1
directory = capnproto-1.0.1
remote = conancenter
method = cmake
[provide]
...

[bruchar1]

1. You don't need to explicitly set the pkg_config_path. Conan set it for for you in the generated native file.
2. The problem I see with your proposal is that conan manages conflicts between packages. If you requires packages A and B from conan, how meson could know whether those packages are compatible or not? Also, conan is optimized to download all dependencies in one big archive, instead of downloading them as separate packages.
3. What I do is that I manage my user options from conan, and inject them in the generated native file. That way, conan knows what dependencies are required or not from those options. And I use custom Python scripts to simplify the calls to conan, meson and ninja.

[mmomtchev]

I am currently exploring the possibility of creating an alternative build system for Node.js native addons with WASM support and I have the same problems - I also need conan with multiplatform support - and I need the possibility of auto-loading native and cross files depending on the environment.

As far as I know there is no way to auto-load native and cross files - the user must build the list one the command line.

Frankly, I wonder if there shouldn't be another super tool that handles conan, meson, builds a list of native and cross files and handles the automatic download of precompiled binaries if these are available. This would be an evolution of @bruchar1 option 3.

[mmomtchev]

Or of course, there is the possibility of loading a cross or a native file from meson itself - but I do not know how feasible it is