Better code reuse for Meson with Meson fragments in Meson language.
#13451
Replies: 4 comments 7 replies
-
|
First problem I found: how to invoke the module several times. I will keep working in ideas to this as things evolve. Probably a module should be something unrelated to meson options (not sure yet) and should be some kind of meson code like a black box that communicates through input and output variables only, in a way that it does not pollute the environment around and that it can be invoked many times. Something along these lines. In this model modules can be used many times and communicate through variables (still flowing everything as an idea, feeedback is very welcome). This is similar to a function, I would say, but without introducing functions themselves. Though it can also add targets, which functions cannot. In this model options are independent of the module, variables are isolated in their own module file: In this model modules only have input and output and do not pollute the outer world, though they can have side-effects such as configuring a file or similar. |
Beta Was this translation helpful? Give feedback.
-
|
@dcbaker what was the functools.partial type thing you were working on? |
Beta Was this translation helpful? Give feedback.
-
|
My thought was having functions where you could pre-set some of the arguments and fill them in later, something like: templ = template(
custom_target,
command : [find_program('generator'), '@INPUT1@', '@OUTPUT0@', '--argument-file', '@INPUT0@']
install : true,
install_tag : 'runtime',
build_by_default : true,
)
x = templ.call(
input : ['my.file', 'other.file'],
output : ['output.file'],
)
y = templ.call(
input : ['file.2', 'file.3'],
output : ['out.file']
) |
Beta Was this translation helpful? Give feedback.
-
|
My design is really about argument re-use |
Beta Was this translation helpful? Give feedback.
-
More ideasSo, taking a look at the initial solution, some evolutionary comments here I have been thinking about: I think an appropriate name for this feature could be 'Meson fragments'. What is a fragment? A fragment is a reusable piece of Meson code in which some input variables are replaced by variables provided by the user and the code is invoked.
The goal of a fragment is to perform actions that currently take quite a bit of boilerplate and cannot be reused except by copy-paste. What are the differences between a fragment and regular functions?A fragment, as functions, has input and output variables and are black boxes. The input variables of a fragment and the output parameters of a fragment can be marked in the Meson fragment. Code reuse can be achieved with regular Meson code. The top-level code is familiar and uses well-known functions from the Meson API. Unlike functions, code fragments can add targets. Use cases in my own codeCreating an installer (fragment consumer side syntax)Create a Meson fragment to use ConanSame idea here. The exposed code in my first comment could use conan, input vars and add a run target like this: Potential syntax for fragment creationIn the next days I will post how the syntax for code fragment creation looks. It is basically the same with some replacement variables marked as inout in some way and with output vars also set. There needs to be a way to mark variables as inout and output. Probably with a new API that declares variables as fragment input vars and output vars and the rest of the code identical to regular Meson code it could be enough. Need to give some thought about it. |
Beta Was this translation helpful? Give feedback.
-
|
muon has support for writing modules in a slightly extended meson syntax: https://github.com/annacrombie/muon/blob/master/doc/script_modules.md Here is the i18n module implemented this way: https://github.com/annacrombie/muon/blob/master/src/script/modules/i18n.meson. Currently these modules must be built-in to muon so must go through the normal review process for any code, but it would be simple to relax that constraint. |
Beta Was this translation helpful? Give feedback.
-
|
@annacrombie thanks for the links I was not aware of those. There has been a lot of heated debate against adding functions in Meson language, including @jpakkane explicit rejection as far as my knowledge goes. I see Meson functions in Muon so I suspect that it would be rejected. That is why I am proposing fragments. Fragments are meant to be isolated pieces that have some differences with functions and are meant to be used coarse-grained. I see fragments as "enclosed normal meson.build syntax" into its own file that can be reused as a function. If instead we have functions, I think there would be a bigger tendency to write special-case functions for each small use of things. That would lead to a proloferation of APIs. Being fragments heavier and more coarse-grained, they are supposed to do something bigger but when you open a fragment you just see normal Meson functions, including the possibility of adding targets. I think this would play in favor of keeping things readable. My experience in CMake is that it becomes programming. In Meson my experience is that things are very uniform (and that is excellent for readability) so the goal in my mind is to enable code resuse but respecting the original APIs and making use of those. If we add functions, the result will be new APIs on top. I still think that for new full APIs a Python module such as qt module or what already exists is the better way. Fragments would be like "pack these 100 lines of code and make them reusable", but those 100 lines of code will not call other fragments. Things stay almost "flat", which contributes to readability. Not even 100% sure if what I am discussing here since it is evolving, but I like the idea of adding minimal syntax, keeping fragments in a single place, keeping the code inside fragments flat and top level as regular Meson syntax and restricting invocation (fragments do not call other fragments) which would lead to a call graph and goes against readability. |
Beta Was this translation helpful? Give feedback.
-
|
Probably the biggest issue to my mind with functions as a generic concept is, indeed, that implementing them in a straightforward way implies adding infinite recursion to the language. I would say that regardless of anything else, the meson maintainers will never change their minds on this. I would not rule out the possibility of meson one day having some form of non-recursive logic reuse. :) Not yet sure what it would look like of course! |
Beta Was this translation helpful? Give feedback.
-
|
@eli-schwartz I agree with Meson maintainers about the recursion stuff. I would not add recursion or arbitraty path loading of things and such. Things should be restricted and located always in the same places. Those make things far easier in the long run. One of the good things from Meson is that we all read the same code such as custom target, executable, test, etc. No tricks. We are all familiar with it. With regular functions things would change that fact: recursion makes reading harder and arbitraty path loading makes discoverability worse. We also find projects in subprojects dir. We find options in meson.options file. You go to a foreigner project and things are obvious. I cannot say the same about other build systems. That is the essence I would like to keep with something like fragments. I would outlaw functions directly in fact: they tend to be more fine-grained, creating new APIs proliferating in all files. They also add new syntax and rules. I think it is simpler to have a piece of current Meson code an say: these are the inputs and this is how to invoke it. Maybe a dcitionary of input and output variables at the top and regular Meson code with a directory for fragments and a single No special cases, no new functions, just Meson code. Nothing. Just that and a directory. And variables shielded from the outer world except for declared input and output variables. For regular functions design, I would still stick to Python modules and on purpose: this makes the barrier to sdd new fine-grained APIs more difficult, discouraging proliferation of unreadable code by others. |
Beta Was this translation helpful? Give feedback.
-
It is trivial to disallow this if desired. On every non-native function invocation simply walk the call stack and check if that function has been called before. This is exactly how meson currently disallows infinite recursion via the
This is actually just a crippled way to emulate arguments and return values to a function where the function is the entire meson source file. |
Beta Was this translation helpful? Give feedback.
-
It could look to you like that, but it has other advantages. For example, it does not add function prototypes and syntax + new rules to the language for that. It also encourages coarse-grained functionality as opposed to the proliferation of new "APIs" in Meson files. I would think of this last thing as a feature, since functions are easily abused for creating new APIs at the slightest chance. So the way I would see it would be: want new functions? Add to a Python module or Meson directly (with its process review, etc) and make them well-known to everyone. Want to avoid boilerplate for your own project(s) but with still readable code for projects and to some extent reusable? Use fragments. The fact that fragments go in a separate file add to the effort to add them to a project, promoting using existing (and understandable) APIs inside those at the top-level. They go in a feagments subdir. Of course they could also be abused as functions in disguise but that design signals them as something heavier. Add random functions and you will end up with DSLs all around instead of regular Meson well-known calls getting nearly equivalent things to what I saw in many CMake projects (minus the recursion). It is easy to creat 10 or 15 functions in a small project with functions that you do not know as a foreign user. But you should not see small projects with 10 or 15 fragments if they are this "heavy" to create, it is just less heavy to call well-known APIs inline. I see that as a feature taking into account that targetting reading uniformity and project structure is a good thing of any Meson project. If you want really involved, fine-grained functions, add them to Meson and promote them as well-known APIs. Otherwise you end up wondering what every call does at every step of reading a Meson file and that goes against Meson readability altogether. At the slightest thing it should be known where a function or fragment is invoked from, clearly. I also suspect that since functions have scope, return types and other stuff, they must be more carefully designed by thinking of new rules. Fragments are just regular code in another file shielded. No function prototypes, not rules for what they return or not. Nothing. Just "use these variables as input" and "use these variables as output". Also, it seems more natural to define targets in those fragments than inside functions. Just my two cents. |
Beta Was this translation helpful? Give feedback.
-
Dear Meson team,
I have been working in my own projects in Meson for a while. FWIW, I would like to discuss an idea that came out from my own usage patterns over time about Meson for better code reuse and show an example of what I am doing currently in the context of a Conan Meson module that is not authored in Python.
I would like to draw a bit of your attention since I am aware of all public discussions (I believe) about adding includes, functions, etc. to Meson. I do not think this is a great idea since it could potentially obfuscate code, but, maybe by reusing much of what we have we can achieve something very close to abstraction and reuse in pure Meson,
but in Meson directly.
I would like to discuss why I think enabling some kind of modules would be a good idea. Since the discussion can be potentially long and involved, I set this topic as a discussion.
The code reuse problem
The real problem with Meson, which is also one of its strengths, is its abstraction level is relatively low but the language very readable: there are no functions, no macros and no way to reuse code easily in the sense of what CMake offers.
However, I am not here to propose any kind of functions or incudes, but to discuss what could be the best way forward to have a better level of code reuse.
For that, I would like to show what I am doing right now and discuss the trade-offs it has to use Python modules in my own opinion compared to authoring some kind of module in Meson.
Meson strengths
I like that Meson does not have too much abstraction when writing files: that makes things easy to read. On the other hand, we cannot make everyone happy and people often demand functions or includes.
Functions, in some way, IMHO, can potentially obfuscate the code when they are not well-known and documented. Arbitrary-path includes break code structure and make path management difficult and project inspection more challenging. I think one of the strengths of Meson is that build files are always called
meson.build, subprojects are always insubprojectsdirectory and options are always inmeson.optionsfiles. You go to a new project and no surprises, no tricks.This makes projects easy to navigate and code, even if a bit verbose at times, easy to follow and understand, together with the readable syntax that Meson has.
I would not change any of those but if I made some changes, it would be in the context of not obfuscating code and still mandating structure into how to organize project structure in general.
Meson modules: current status
Meson currently has a set of modules, written in Python. Writing modules in Python is the way to go if you want new features into Meson.
Pragmatically speaking, I think that having to go to Python and the Meson repository just to add reusable code is a far cry for most developers that are just interested in a low-cost mechanism to reuse code without resorting to copy-paste and similar.
Most people are not going to author a Python module and go through all the process to make it into a Pull Request, get reviews, having the thumbs up from maintainers, etc. This is time-consuming and in many occasions not worth the effort.
However, the need for code reuse remains.
Pure meson module (renamed to
Meson fragment, better syntax pending for input/output vars) what I did and the patterns that emerged.This is how I am currently implementing Meson "modules" on my own side. This is just a discussion on how to move things forward. I am not proposing anything beyond feedback at this point.
A Module, the way I am doing it, provides some
meson.optionsto the user. Currently, in order to use those options, I paste them in the top-level directory.Good things about having a "module" with Meson code:
A Module has, in my own implementation:
The good thing about modules is that they are regular Meson code, without new abstraction mechanisms. Code is clear to read once you open it, because Meson is easy to read.
A Module lives always in
$project_root/meson/modules/modulenamein my case, in$project_root/meson/modules/conan. The modulename directory contains a regularmeson.buildfile.My Conan "module" (renamed to
Meson fragmentsin the coming comments)My conan module has all its documentation at the top, so please read and check, and follow after reading:
How I use the Meson "module"
At the top-level or any level of the project, I can use this module by doing:
So basically, this has achieved code reuse, maybe with a couple of caveats:
The module is basically data-driven, which is a good thing:
In this sense, I think it is nicer than polluting your code with real user-defined functions around, though it could be seen as half-equivalent in some way. The usage from outside is intended to be through those two input mechanisms and and output variable mechanisms (probably even files actually...?), so the scope is well-defined.
Module reuse and potential improvements
I can imagine a github repo (that I could even open myself!) and systematize these Meson modules without any new language features to be able to reuse code logic for this Conan module, installers and others.
This is not a proposal, just a discussion, though I think that, with minimal changes, we could have this mechanism standardized in some ways:
Maybe input variables could be marked in some way? Not sure it is even needed. It works as it is currently but they are not marked in any way.
This "module" method already works, just by making it minimally prettier and without messing with Meson rules or very elaborate stuff, it is perfectly usable IMHO.
I think this has the advantage that modules can be collected and reused for projects more easily. It also has the advantage that it uses well-known features and mechanisms yet it does not add new rules or anything similar to the language itself, just a couple of ways to get the options and better encapsulation.
What do you think? Problems? Improvements?
Beta Was this translation helpful? Give feedback.
All reactions