mesonpy embeds random path into .o files for debug build

[bmwiedemann]

While working on reproducible builds for openSUSE (sponsored by the NLnet NGI0 fund), I found that various python packages (python-pandas, python-scikit-learn, python-scipy, python-contourpy, python-gobject) have variations in every build.

I am pretty certain this comes from

meson-python/mesonpy/__init__.py

Line 965 in 04e0066

build_dir = ctx.enter_context(tempfile.TemporaryDirectory(prefix='.mesonpy-', dir=source_dir))

but don't know how to best make this deterministic.

Is there a variant of tempfile.TemporaryDirectory that does not add randomness? I tried with pathlib.Path(os.path.join(source_dir, '.mesonpy')) but that lacks the automatic cleanup.

Here is a typical diff:

--- old /usr/lib/debug/.dwz/python-pandas-2.2.2-1.1.x86_64 (objdump)
+++ new /usr/lib/debug/.dwz/python-pandas-2.2.2-1.1.x86_64 (objdump)
@@ -14923,7 +14923,7 @@
  0010 01010101 081e3c64 777a3e00 2f757372  ......<dwz>./usr
  0020 2f737263 2f646562 75672f70 616e6461  /src/debug/panda
  0030 732d322e 322e322f 2e6d6573 6f6e7079  s-2.2.2/.mesonpy
- 0040 2d6e7777 6b673134 74002f75 73722f6c  -nwwkg14t./usr/l
+ 0040 2d73615f 6761376e 33002f75 73722f6c  -sa_ga7n3./usr/l
  0050 69623634 2f676363 2f783836 5f36342d  ib64/gcc/x86_64-
  0060 73757365 2d6c696e 75782f31 342f696e  suse-linux/14/in
  0070 636c7564 65002f75 73722f69 6e636c75  clude./usr/inclu
@@ -14965,7 +14965,7 @@
  02b0 2f6e756d 70792f64 61746574 696d6500  /numpy/datetime.
  02c0 2f757372 2f737263 2f646562 75672f70  /usr/src/debug/p
  02d0 616e6461 732d322e 322e322f 2e6d6573  andas-2.2.2/.mes
- 02e0 6f6e7079 2d757868 63736133 66002f75  onpy-uxhcsa3f./u
+ 02e0 6f6e7079 2d713166 36397136 61002f75  onpy-q1f69q6a./u
  02f0 73722f69 6e636c75 64652f70 7974686f  sr/include/pytho
  0300 6e332e31 31002f75 73722f69 6e636c75  n3.11./usr/inclu

[eli-schwartz]

You are "holding it wrong":

https://gitweb.gentoo.org/repo/gentoo.git/tree/eclass/distutils-r1.eclass#n1297

[thesamesam]

The line above is:

meson-python/mesonpy/__init__.py

Line 964 in 04e0066

if build_dir is None:

... so set build dir if you don't want one chosen for you.

[bmwiedemann]

So far our build-description just says %pyproject_wheel so not sure where to set builddir without patching every individual package. Maybe in our python-rpm-macros.

A more deterministic default would still be appreciated. e.g. scipy is listed on https://github.com/mesonbuild/meson-python/blob/main/docs/projects-using-meson-python.rst as "known to be adhering to best practices" but seems to also not specify a builddir.

Edit: docs suggest python -m build --wheel -Cbuild-dir=build

Now I got a reproducible build with

%pyproject_wheel --config-settings build-dir=mesonpybuild .

However, this is not so reassuring:
https://meson-python.readthedocs.io/en/stable/how-to-guides/build-directory.html

Warn that user-configured build directories are “use at your own risk”

[bnavigator]

The "how-to-guides" link leads to an older version.

https://mesonbuild.com/meson-python/reference/config-settings.html does not mention the at-own risk.

We have a clean build directory for every package (and python flavor) built by rpmbuild, so I think we can handle the risk.

[eli-schwartz]

So far our build-description just says %pyproject_wheel so not sure where to set builddir without patching every individual package. Maybe in our python-rpm-macros.

That sounds like an excellent question to ask the rpm or other OpenSUSE developers. I don't think meson-python has any opinion or indeed knowledge at all about what rpm-specific mechanisms rpm is using to execute meson-python or pass options to meson-python.

A more deterministic default would still be appreciated. e.g. scipy is listed on https://github.com/mesonbuild/meson-python/blob/main/docs/projects-using-meson-python.rst as "known to be adhering to best practices" but seems to also not specify a builddir.

It seems you are still confused about the option then? Why would scipy specify a builddir when it is the job of the build orchestration to decide whether a builddir makes sense? The default is carefully chosen -- the option is to "enable reproducibility and disable protections against clobbering manual user-configured build directories for people who don't use distro automation and just pip install .

However, this is not so reassuring: https://meson-python.readthedocs.io/en/stable/how-to-guides/build-directory.html

Warn that user-configured build directories are “use at your own risk”

Even for out of date documentation (which admittedly should be deleted) that is no longer referenced, it is marked as a TODO. It is vagueness upon vagueness. You shouldn't just take that with a grain of salt, you should take it with a whole bucket of salt.

[rgommers]

UX-wise there's a tradeoff here. With the extra tmpdir that meson-python creates:

1. The build always happens in a fresh directory and hence there's no risk of previous build settings being picked up from an earlier build.
2. Apparently there's non-reproducibility being introduced, which is annoying - repeated builds should yield identical results by default in the same build environment, ideally without extra build flags.

(1) seems more critical than (2) and has a much larger number of users, so our current choice does look right. However, the other question I'd have is what exactly gets embedded in those .o files - since if it's absolute paths, then that's anyway bad for build reproducibility (they'd be reproducible on the same machine or in the same CI config, but not otherwise). Could those be problems in pandas, SciPy & co?

I currently don't have much bandwidth, but if there's a fixable problem here I'd certainly want to prioritize fixing that for SciPy.

[rgommers]

I did a test and cannot actually reproduce the problem here with SciPy. Procedure:

1. export SOURCE_DATE_EPOCH=166887191
2. python -m build -wnx # in scipy repo, current main - build wheel without build isolation
3. mv dist/scipy-1.15.0.dev0-cp312-cp312-macosx_14_0_arm64.whl dist/scipy_bld1.whl
4. python -m build -wnx
5. mv dist/scipy-1.15.0.dev0-cp312-cp312-macosx_14_0_arm64.whl dist/scipy_bld2.whl
6. diffoscope dist/scipy_bld1.whl dist/scipy_bld2.whl

diffoscope says everything is identical.

[eli-schwartz]

It's the debug info in unstripped binaries.

[rgommers]

Ah that makes sense, thanks. It's not clear from the issue description or the links given there though. @bmwiedemann can you please confirm that all your regular package builds are fine, and it's only building debug packages that is showing non-reproducibility?

[bmwiedemann]

Yes, building without debuginfo is reproducible.
However, the default in openSUSE's Open Build Service is to always build with debuginfo and automatically strip it into a separate -debuginfo sub-packages at the end (maybe here). This leaves a hash of the debuginfo in the stripped .so file for gdb to find the right data, which is what makes the binary vary. In objdump diffs this shows up as a "shstrtab" section.

[rgommers]

This leaves a hash of the debuginfo in the stripped .so file for gdb to find the right data, which is what makes the binary vary.

The build ID hash depending on an absolute path to a build directory that doesn't exist at runtime is a design problem in debuginfo it looks like. While you can work around it by using --build-dir as pointed out above, this is an issue that ideally shouldn't occur to begin with. I think it's described in https://fedoraproject.org/wiki/Releases/FeatureBuildId:

The program /usr/lib/rpm/debugedit is used in rpm's separate debuginfo creation. It modifies DWARF data to replace the build-time absolute directory names with consistent names. This makes the installed debuginfo's source references usable, and it makes for reproducible rpm builds from identical constituents to produce identical binaries. The build ID computed by ld was affected by the name of the rpm build directory, so it will differ between two otherwise identical builds that used a different $RPM_BUILD_ROOT.

My patch adds the -i option to make debugedit recompute the build ID based on the contents of the file after the transformation. It also prints out the build ID bits in hex.

This is better than using --build-dir I think, since you don't want absolute paths to affect the hash and should apply independent of what build system or Python build backend you're dealing with.

[bmwiedemann]

Another way out could be the gcc -fdebug-prefix-map=old=new option.

[rgommers]

That looks like it's in the right direction too indeed. Not sure it can be made to work though, since you have to pass the name of the old directory, and if that's a tmpdir whose name you don't know beforehand (which will be the case for generated source files at least), what do you pass? Perhaps a good idea to use in combination with --build-dir though, if you end up going that way?

I'll include the description for fdebug-prefix-map=old=new from https://gcc.gnu.org/onlinedocs/gcc/Debugging-Options.html for future readers:

When compiling files residing in directory old, record debugging information describing them as if the files resided in directory new instead. This can be used to replace a build-time path with an install-time path in the debug info. It can also be used to change an absolute path to a relative path by using . for new. This can give more reproducible builds, which are location independent, but may require an extra command to tell GDB where to find the source files. See also -ffile-prefix-map and -fcanon-prefix-map.

[bmwiedemann]

since you have to pass the name of the old directory, and if that's a tmpdir whose name you don't know beforehand (which will be the case for generated source files at least), what do you pass?

mesonpy knows the name of the tmpdir, so could it add this option to CFLAGS (and CXXFLAGS)?