Interactive error fixing

[projekter]

When configuring a large project, probably with lots of subprojects (I'm currently trying to compile gstreamer), often some unresolved dependencies show up. Usually they can be fixed very easily by just a single call to a package manager. Then, the configuring process starts from the beginning, until the next missing dependency is found. A lot of time is wasted on things that essentially were already done.
Of course, it is not Meson's job to solve the dependencies. But would it be possible to allow for some "interactive" fixing - for example, do not terminate, but instead either provide an option

• for a subshell, so that the user can execute some command that is supposed to fix the dependency, then retry the last dependency resolution
• or just provide a prompt to either exit or retry, then in some other process/shell, the user can do whatever is necessary to resolve the issue
  and the meson continues where it was.

Something that may be an issue here is that environment variables might have changed and should be re-read when retrying. Would this interfer with previous dependency resolutions? If they were turned into absolute paths at their time of resolution, this should not cause any issues.

[xclaesse]

That's something I've been wanting too, but it could turn into a can of worms, I'm not sure... I know @jpakkane is against interactive mode, but IMHO with an explicit option --interactive it sounds good to me. Maybe a simple "Dependency foo is missing. Retry? [Y/N]".

environment variables might have changed and should be re-read when retrying

Why would they change? I think the main use-case here is a missing pkgconfig file. User installed the -dev package and Meson can retry the dependency lookup.

[projekter]

You're probably right that it is unlikely that PATH variables should change. This really depends on how "invasive" the resolution is. It is probably much more likely on Windows systems than on *nix; but since on Windows, you'd usually have to do much more than just a quick install command, it is also much acceptable that the meson should re-run from the start. So yeah, change in environments should not occur very frequently and it is reasonable to then demand just restarting the configuration.
But then, this is really something that should be minimally invasive. I think an --interactive switch would be a good idea, probably with three options: disabled (as it is now), enabled (so always prompt when something is missing) or required (only prompt for required packages/programs/headers).

[xclaesse]

The thing that makes me a little bit nervous is if a dependency is checked twice (e.g. in different subproject) and the 1st you say N (or wasn't asked because it was not required) and the 2nd you say Y. You end up with inconsistent build. If later your do a reconfigure it will have that dependency in the first case where you previously said N.

That's the kind of situation why I said this could turn into a can of worms. Maybe after an interactive configuration, it should automatically reconfigure again in non-interactive mode.

[projekter]

Indeed, so either Meson keeps track of a list of all the dependencies for which a user prompt was already received, so that every dependency is only asked for once. Or in the end it just prints a warning message that a full reconfigure is highly recommended or really does it automatically. I'm really fine with any of these solutions (though maintaining a global state would be pretty neat to avoid too much prompting), as they all make sense and help improving the workflow.

[eli-schwartz]

At least on Linux, environment variables cannot be changed from a sub- or parallel process.

This is either good, as it doesn't interfere with previous dependency lookups, or bad, because you can't actually install a dependency and continue if that needs to update the environment too. For example, a dependency whose pkg-config file is installed to a custom $PKG_CONFIG_PATH.

I can't remember offhand if restarting a failed meson setup utilizes the cache? It does for automatic reconfiguring on updates to meson.build, so that usually saves a lot of forking for dependency lookup and compiler checks.

Note that many projects list their dependencies in README.md as e.g. "run this command to install all dependencies on Debian, or this other command to install all dependencies on Fedora". This eliminates the guesswork.

[eli-schwartz]

Windows environment variables are looked up in the registry, IIRC. Update those environment variables, and the next time any program anywhere looks up an environment variable, it sees the new version.

Linux environment variables are:

• per process
• inherited from, cloned and forked from the parent process

You can change environment variables of a running process only by being that process and modifying its own state, or by attaching a debugger to that process and using debugger magic to edit that process's memory. So in that sense, sure you can modify environment variables of a running process on Linux.

But also no, because attaching a debugger to the pid of the running Meson isn't exactly an intended workflow. :p

[gregsskyles]

Actually, environment variables in Windows operate in much the same way as you describe for Linux. The are stored in the registry, yes, but only read during process creation. Once a process is running, the registry version of a variable can change, without effect on the running process. There are APIs to change an environment variable from within the process, but those changes disappear when the process dies (unless you write them back to the registry).

[eli-schwartz]

I think that doesn't really contradict what I said, though? I suspected that was the case, wasn't sure, and didn't state one way or the other.

Considering the conversation is about changing environment variables from a spawned shell prompt as a child process, though, it's actually worse for this discussion for Windows to do the expected thing and freeze the variables at process creation. Because that means Meson won't see them!

Tools which Meson executes would see them, so you'd end up in an inconsistent state where this whole proposed idea only works on Windows and only if the dependency is looked up via e.g. pkg-config via a variable Meson doesn't force set (I don't remember offhand, but I think it will explicitly set PKG_CONFIG_PATH to the current setting, even if empty, but it won't actually try setting _LIBDIR), but not for something Meson handles internally, such as the boost root.

The fact that it's stored in the registry is relevant, because Meson could have inconsistent Windows-specific code to read the registry by hand and update os.environ after this theoretical shell prompt. I'm not inclined to do that though.

[projekter]

It is not necessarily the case that newly started processes (CreateProcess) would see changed environment variables. Usually, on Windows, a process inherits the environment variables of its parent process - so if Meson doesn't see them, a child process also won't see them, regardless of whether another, earlier child process did some modifications. However, this is not always the case (else, no environment changes would ever be noticed), for example processes started using the Explorer (or, in terms of API, ShellExecute) will always fetch the current set of environment variables from the registry; and the parent process can also use additional parameters in CreateProcess to create custom environment variables just for the child process.
So this means that this is actually pretty safe and also fairly consistent with the *nix side.

It is probably best to just forget about my idea with the environment variables at the moment. Maybe if there are complaints in the future, this can be picked up again. It's not the case that it is really impossible to achieve, if we wanted to mimick "open console, run Meson, fail, install dependency, run Meson again (same console)" with just a single run, then this would correspond to the following scenarios:

• If the dependency resolution is done via an extra process, this will not have any impact on the environment of the console, and hence also not on Meson's second run. Therefore, also in interactive mode, nothing would have to be considered. This holds true on *nix and Windows.
• However, if the resolution is done via a bash script, this can potentially change the environment variables of the console and therefore of the second run. This also holds true on both *nix and Windows; but while this is proabably the expected scenario on *nix, it is very unlikely on Windows that "just" a batch script is responsible for such a task.
  If Meson just asks "retry," and the actual resolution is whatever, but unknown to Meson then, there is nothing that can be done on *nix; on Windows, we could in principle read the global environment configuration, but this would be pretty inconsistent.¹
  If Meson allows to enter a subshell to do the resolution, this subshell could upon exit inform Meson about its environment, and this could then be loaded. On *nix, the subshell (say, bash), could for example be started with bash --init-file <(echo "trap 'export -p > /tmp/myenv.sh' EXIT; echo 'If you are done with the resolution, type \'exit\'.'") (better some unique temporary name), the output would then have to be parsed in Python and the environment set appropriately. Not impossible, but a bit running before walking.

¹ Since if the installation was not done with a batch script, the "new" global environment would actually not be known to Meson even in the two-runs scenario unless the console itself was restarted. And since the console itself might have changed environment variables, so even Meson's first run might not start with the global configuration - hence, resetting to it would be extremely inconsistent. It would make sense to perform a diff and merge on the environment variables, but this is far more complicated than anything that would happen in the two-runs scenario.

[gregsskyles]

No, @eli-schwartz not trying to contradict you at all, more meaning to say, as @projekter does "... also fairly consistent with the *nix side." I do agree with Benjamin that it could be a bit fragile to try to account for changes to environment variables in this proposed interactive mode at first.

[xclaesse]

Maybe an alternative would be print a summary of not-found optional dependencies at the end of the first configure. Then you can install them and reconfigure with -Dauto_features=enabled?

[projekter]

Well, as I said I finished configuring gstreamer, so this is now purely theoretical; my purpose with this discussion was not to complain about a particular project, but to point out a useful and (I think) rather easy-to-implement feature.
So, generically speaking (for any kind of project), if the configure script always checks for all the packages/libraries/dependencies and then enables the feature if it was found, this would be a good option. If some of the checks are only conditionally executed (for example, assume feature A did not find a dependency, so it is disabled; now feature B requires feature A plus some dependencies, but since A is disabled, it will never check; or think of lazy checking that disables as soon as the first missing dependency is detected), then something is missing.
But yes, this would probably also be a good idea and - I don't know if there is a particular log for this yet - could easily be achieved by grep-ing over the configure log.