UX for constraining build-time and runtime dependencies

[rgommers]

There is a practical UX problem with pyproject.toml that we need to solve. A short description would be:

• Any project that builds against the numpy C API has to deal with API and ABI compatibility (see the numpy docs on that for details).
• as a consequence, the dependency specification for local development is different from the one needed for uploading an sdist to PyPI:
  • for local development, you want loose dependencies, e.g. numpy >= 1.19.5 (both a build and runtime dep)
  • in sdist released to PyPI you want: oldest-supported-numpy (build dep metapackage, contains == pins per platform) and numpy>='buildtime_dep,<1.25.0' # should be N+3whereN is current minor version.

That is obviously a complex set of dependencies to express in pyproject.toml, and we can't have two sets in there at the same time. The solution @FFY00 and I just discussed is:

1. On the main branch, use the loose development dependencies (numpy>=1.19.5)
2. In the CI job(s) that produce artifacts for release to PyPI, ensure that the lowest supported dependencies (which may be platform-dependent) are installed.
3. In the release branch, replace the dependencies with the build-time deps needed for uploading to PyPI.
4. Have support in mesonpy for the various kinds of pinnings people need for releases and API/ABI compatibility.

A rough sketch of how (4) could look:

[build-system]
requires = [
  "numpy >= 1.19.5",  # replace with (e.g.) `oldest-supported-numpy` in a release branch
]

[tool.mesonpy]
runtime_pinnings = [
  # if build version detected is 1.20.3, replace with `>=1.20.3` (while preserving the upper bound,
  # so end result would be for example `'>=1.20.3,<1.25.0')
  "numpy = 'pin_compatible'",
  "pythran = 'bugfix_only'",  # if build version detected is 0.10.0, override the build dependency with `'>=0.10.0,<0.11.0'`
  "torch = 'exact'",  # runtime version must equal build-time version (PyTorch has no ABI stability guarantees)
]

Note that this feature is needed for some other packages than just NumPy too (basically every package offering a C/C++ API has to deal with this), but NumPy alone is important enough - every package containing even a single Cython extension which uses NumPy has to deal with this. However, it's probably still a bit too specific to want native support in pyproject.toml via a standard that all tools must implement. Hence the choice to put this in mesonpy.

[FFY00]

I have let this sink in a bit more and thought about it.

The only slight worry I have with this approach, is that the wheels will not be necessarily reproducible. I don't think it's a big issue, as anyone serious about reproducible wheels should be reconstructing the original build environment anyway¹, and this won't be in issue there. Still, I feel very slightly uneasy about the built artifacts being highly dependent on the environment ended up being resolved by Pep 517 builders. Just something worth nothing IMO.

Other thing, I think we should be adding a local tag to the version (eg. 1.2.3+numpy3.2.1) to easily identify the constraints from the build dependencies.

---

The main design decision here would be how to specify the locked version, I would really like to gather some feedback from other people here.

A similar problem, and its solution, can be found at https://www.python.org/dev/peps/pep-0440/#compatible-release.

My initial proposal would be:

...

[tool.mesonpy.pin-build-dependencies]
numpy = '*.*.*'

And the tricky part, the patterns:

1. Simple approach:
   Use a placeholder character, like * (eg. *.*.*), other option would be ^ (eg. ^.^.^).

2. Full-on regex:
   Allow users to specify regex patterns, which will be run against the version string of the dependency (eg. ^[0-9]*.[0-9]*.[0-9]).

All the examples would pin the first three version number (eg. for 1.2.3.4.5 we would lock 1.2.3).

Anyway, none of this is trivial to implement, and there isn't much difference in difficulty either, I think regex might be a bit easier? So, I think we should go with what's easier for users, unless that proves itself problematic when implementing.

That said, it is worth to point out that if we are constraining the dependencies on the fly, we need to put dependencies in dynamic and move it to the backend config. This is not optimal, but unfortunately PEP 621 does not let us keep the dependencies setting in the project table if we mark it as dynamic.

...

[project]
...
dynamic = [
  'dependencies',
]

[tool.mesonpy]
dependencies = [
  'numpy~=3.2.1',
]

[tool.mesonpy.pin-build-dependencies]
numpy = '*.*.*'

Footnotes

1. And there has been a push from the community towards this, with PEP 665, though it has been rejected due to it not really targeting sdists. I think the proposal might revived sooner or later, hopefully with also sdists in mind. Anyway, I think this shows us that there are bigger needs here and that it is perfectly reasonable to not concern us too much with this issue from the build system side. For reference, other packaging ecosystems/systems, like Arch Linux will still be able to achieve full package reproducibility, so this really seems like a PyPI / PEP 517 issue. ↩

[rgommers]

The only slight worry I have with this approach, is that the wheels will not be necessarily reproducible. I don't think it's a big issue, as anyone serious about reproducible wheels should be reconstructing the original build environment anyway1, and this won't be in issue there. Still, I feel very slightly uneasy about the built artifacts being highly dependent on the environment ended up being resolved by Pep 517 builders. Just something worth nothing IMO.

Agreed, need to have a reproducible environment for building. I think this topic is something we're going to have to teach people a bit about anyway (I've started some write-ups). It is absolutely untrue that PEP 517 will give you reproducible builds when you have compiled extensions. The illusion that they are reproducible is more harmful than helpful imho.

Other thing, I think we should be adding a local tag to the version (eg. 1.2.3+numpy3.2.1) to easily identify the constraints from the build dependencies.

Hmm, I'm not sure I like messing with version strings, that seems unjustified (and what do you get when you have say five such dependencies?). A project like SciPy will already have a complex version string - from the latest nightly: scipy-1.9.0.dev0+1215.a0e129f-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

My initial proposal would be:

Interesting - yes there's a lot of design flexibility here. I have no clear opinion yet. Note that conda-forge implements this in a reasonable form, looking at its pin_compatible and run_exports may be useful: https://conda-forge.org/docs/maintainer/pinning_deps.html. At first sight, seems easier to understand that regexes (and with more semantic meaning, e.g. ABI compatibility).

That said, it is worth to point out that if we are constraining the dependencies on the fly, we need to put dependencies in dynamic and move it to the backend config. This is not optimal, but unfortunately PEP 621 does not let us keep the dependencies setting in the project table if we mark it as dynamic.

Ah, good point. That may be a problem - aren't other tools supposed to be reading that dependencies table? I'm not sure what the implications would be exactly.

[rgommers]

A similar problem, and its solution, can be found at https://www.python.org/dev/peps/pep-0440/#compatible-release.

I looked at this again, and want to point out that ~= is quite a bit simpler than what we need. Using more operators doesn't seem ideal to me, that will be more or less incomprehensible to most devs without looking up the syntax.

Your example pyproject.toml snippet is the wrong way around (it pins runtime dependency and then finds a compatible build dependency). The logic we want instead is to specify the version of the build dependency, and then adjust runtime dependency dynamically (but not with ~= usually, we need to allow a larger range at runtime). I think this will be generically true for packages where ABI matters.

[pradyunsg]

(this is a drive-by comment, sorry!)

Regarding 4, a couple of ideas...

To go the route of mirroring the existing style from conda-build, where you are "calling" a function.

[tool.mesonpy]
runtime-pins-from-build-dependencies = [
  "compatible(numpy, *.*)",
  "bugfix_only(pythran)",
  "exact(torch)",
]

Alternatively, lean into structured TOML data:

[tool.mesonpy.runtime-pins-from-build-dependencies]
numpy = {strategy="compatible", constraint="*.*"}
torch = "exact"

That is obviously a complex set of dependencies to express in pyproject.toml, and we can't have two sets in there at the same time.

Well, depends on how/when you think those two should be composed. Basically, I'm imagining something like...

[tool.meson]
release-dependencies = [
   "oldest-supported-numpy",
]

(a) If it's OK to have an environment variable that is always set during release... That'll let you have dynamic dependencies via a get_requires_for_build_whatever that reads a list and tacks it on for release builds. When building the sdist, write the additional build dependency in build-system.requires, if the variable is set. (pick your opt-in/opt-out semantics)

(b) It's also reasonable to skip dynamic build dependencies and instead rely upon the build "flow" difference between pip install's consumer-facing behaviour of sources -> wheel vs build's ¹ publisher-facing sources -> sdist -> wheel. When building the sdist, write the additional build dependency in thee pyproject.toml build-system.requires always.

This, of course, means that you're assuming that all sdist builds are eventually-public and I reckon that's reasonable.

IIUC, between a strategy for pinning runtime numpy based on the build environment and, a way to ensure oldest-supported-numpy be added as a build dependency for release builds will be sufficient to get the behaviours you want?

FWIW, you could even squish all of this into a single table...

[tool.mesonpy.runtime-pins-from-build-dependencies]
numpy = {strategy="compatible", constraint="*.*", release-requires=["oldest-supported-numpy"]}
torch = "exact"

Other thing, I think we should be adding a local tag to the version (eg. 1.2.3+numpy3.2.1) to easily identify the constraints from the build dependencies.

What you're describing here feels to me like a poor man's substitute for an SBOM, for describing the build environment. There's no need to try and squish that into the local version label IMO.

It might reasonable to just dump the relevant build environment information into a file in the sdist/wheel, and use that... instead of trying to squish things into the local version label. And, if you want the protections that a local version label provides against a PyPI upload, put a sentinel value in there when you generate the non-release variants.

Footnotes

1. A future pip build's semantics will match build's. ↩

[rgommers]

Thanks for the input @pradyunsg!

Alternatively, lean into structured TOML data:

TIL there's such a thing. I don't have a real preference here - both syntax choices work, and are about equally readable. I'm not quite sure what constraint="*.*" means though - why do we need this? Or in other words, what can that be aside from *.*?

If it's OK to have an environment variable that is always set during release...

I'd prefer to avoid that, because it's going to be one of those things people forget whenever they build outside of the release CI jobs.

(b) It's also reasonable to skip dynamic build dependencies and instead rely upon the build "flow" difference between pip install's consumer-facing behaviour of sources -> wheel vs build's 1 publisher-facing sources -> sdist -> wheel. When building the sdist, write the additional build dependency in thee pyproject.toml build-system.requires always.

I like this option better. A little heretical for a PyPA/Pip dev to suggest it perhaps, but hey I'll take it:)

IIUC, between a strategy for pinning runtime numpy based on the build environment and, a way to ensure oldest-supported-numpy be added as a build dependency for release builds will be sufficient to get the behaviours you want?

Yes, I believe it is. That said, it's an additional improvement over having to make manual changes in release branches, and it's currently of secondary importance compared to pin_compatible. So I'd probably start with implementing just numpy = {strategy="compatible", constraint="*.*"}, and then leave release-requires= for after that.

It might reasonable to just dump the relevant build environment information into a file in the sdist/wheel,

We partly do that anyway, for example the detected BLAS/LAPACK version gets written to a generated .py file and exposed in numpy and scipy through a show_config() function. Extending that to dump the full configure output and environment information is quite useful - I was already planning to do something like that.

[pradyunsg]

I guess I used constraint instead of min_pin out of being uninformed. The latter is a better name anyway.

https://docs.conda.io/projects/conda-build/en/latest/resources/variants.html?highlight=pin_compatible#extra-jinja2-functions

[rgommers]

I guess I used constraint instead of min_pin out of being uninformed. The latter is a better name anyway.

Ah, this makes sense now, thanks. And it's another choice to make. We don't just need min_pin but also upper_bound (e.g., numpy should be pinned to < 1.most_recent_minor_version + 3). This could all be done in a single expression:

numpy = {strategy="compatible", min_pin=x.x.x, upper_bound=1.27.0}  # expressing it as `max_pin=x.x+3` would be great, conda-build doesn't allow this 

or it could be done separately like I originally suggested:

[tool.mesonpy.runtime-pins-from-build-dependencies]
# note that we don't need `min_pin`, because >=1.19.5 is already specified in the build deps
numpy>='pin_compatible,<1.27.0' 

I think in the end all of these are roughly equivalent, it's mostly a matter of preferred syntax (what's easiest to read & parse). I suspect structured TOML is best from a parsing perspective.