Using Meson to install a nested subdirectory into a specific namespace

[adam2392]

Hi,

Say I have a directory structure, where there are some Cython files.

sktree/
&emsp _lib/
&emsp &emsp meson.build
&emsp &emsp sklearn/
&emsp &emsp &emsp sklearn/
&emsp &emsp &emsp &emsp tree/
&emsp &emsp &emsp &emsp &emsp _tree.pxd
&emsp other_file.py

and say my sktree/_lib/meson.build file looks like this:

# install tree/ submodule
extensions = [
  '_tree',
]

foreach ext: extensions
  py3.extension_module(ext,
    cython_gen_cpp.process('./sklearn/sklearn/tree/' + ext + '.pyx'),
    c_args: cython_c_args,
    include_directories: [incdir_numpy,],
    install: true,
    subdir: './sklearn/sklearn/tree',
  )
endforeach

where does this get installed? I tried adding an import statement like from sktree._lib.sklearn.sklearn.tree._tree cimport DOUBLE and it gives me an error: 'sktree/_lib/sklearn/sklearn/tree/_tree.pxd' not found.

My next follow-up question is can we install any arbitrary nested files into a specific namespace, so it's importable from that namespace? E.g. to allow from sktree._lib import stuff?

[eli-schwartz]

subdir: './sklearn/sklearn/tree',

This means it should be installed into the wheel or into site-packages as sklearn.sklearn.tree.{ext}.

My next follow-up question is can we install any arbitrary nested files into a specific namespace, so it's importable from that namespace? E.g. to allow from sktree._lib import stuff?

You mean like .py files? You probably want to do this with py3.install_sources('stuff.py', subdir: 'sktree/_lib') to get something that you can then run from sktree._lib import stuff.

[eli-schwartz]

This error comes at build time from cython, before the subdir for installation actually comes into effect.

It's basically saying that based on cython's current understanding of the import path for pxd files, it can't find the pxd import you're asking for.

I think you can add -I arguments to cython itself, which cause it to add those directories to the search path for pxd files?

@rgommers may know more.

[adam2392]

Hmm that sounds weird to me because the current meson file builds correctly:

fs = import('fs')
if not fs.exists('sklearn/README.rst')
  error('Missing the `sklearn` fork submodule! Run `git submodule update --init` to fix this.')
endif

# install tree/ submodule
extensions = [
  '_tree',
  '_utils',
]

foreach ext: extensions
  py3.extension_module(ext,
    cython_gen_cpp.process('./sklearn/sklearn/tree/' + ext + '.pyx'),
    c_args: cython_c_args,
    include_directories: [incdir_numpy,],
    install: true,
    subdir: 'sktree/_lib/sklearn/tree/',
  )
endforeach

# install test file
extensions = [
  # 'stuff',
]

foreach ext: extensions
  py3.extension_module(ext,
    cython_gen_cpp.process('./' + ext + '.pyx'),
    c_args: cython_c_args,
    include_directories: [incdir_numpy,],
    install: true,
    subdir: 'sktree/_lib/',
  )
endforeach

But as soon as I uncomment 'stuff', where the only line inside the .pxd file is:

from sktree._lib._tree cimport DOUBLE_t

it errors out. Moreover, my build directory actually shows the sktree._lib._tree:

So doesn't the first part compiling correctly indicate that the Cython files were already installed into the subdir specified?

[rgommers]

@adam2392 it looks like you are installing the extension modules (.so's), but your screenshot nor your code show any trace of a .pxd file. Cython does not generate those automatically; if you use from sktree._lib._tree cimport, then right next to _tree.cpython-39-darwin.so there should be a _tree.pxd file. In case that file is present in your source dir, you'll want to copy it over to the build dir with fs.copyfile. And the same for __init__.py files. This is a little painful unfortunately, because the cython import system depends on the whole file tree. There's examples in SciPy of how to deal with this.

[adam2392]

Ah I see... I realize then there are two issues:

1. a submodule installs a whole repo, of which the root does not contain __init__.py, which presents challenges for Cython
2. the PXD issue you mentioned now.

For sake of completeness, I went with the following solution: I override the meson.build command in spin and do some behind-the-scenes moving of folders to make sure that sktree/_lib/sklearn/ is now a proper subdirectory with __init__.py and also then I can change the relevant meson.build to build with the subdir: 'sktree/_lib/sklearn/tree'

In case ppl have this issue in the future, here is my complete .spin/cmds.py file:

@click.command()
@click.option("--forcesubmodule", is_flag=True, help="Force submodule pull.")
def setup_submodule(forcesubmodule=False):
    """Build scikit-tree using submodules.

    git submodule update --recursive --remote

    To update submodule wrt latest commits:

        git submodule update --init --recursive --remote
        git add -A
        git commit

    This will update the submodule, which then must be commited so that
    git knows the submodule needs to be at a certain commit hash.
    """
    commit_fpath = "./sktree/_lib/sklearn/commit.txt"
    submodule = "./sktree/_lib/sklearn_fork"
    commit = ""
    current_hash = ""

    # if the forked folder does not exist, we will need to force update the submodule
    if not os.path.exists("./sktree/_lib/sklearn/") or forcesubmodule:
        # update git submodule
        util.run(["git", "submodule", "update", "--init", "--force"])
    else:
        # update git submodule
        util.run(
            [
                "git",
                "submodule",
                "update",
                "--init",
            ]
        )

    # get the commit hash if the commmit file exists
    if os.path.exists(commit_fpath):
        with open(commit_fpath, "r") as f:
            commit = f.read().strip()

    # get revision hash
    current_hash = get_git_revision_hash(submodule)

    print(current_hash)
    print(commit)

    # if the commit file doesn't exist or the commit hash is different, we need
    # to update our sklearn repository
    if current_hash == "" or current_hash != commit:
        util.run(
            [
                "mkdir",
                "-p",
                "./sktree/_lib/sklearn/",
            ],
        )
        util.run(
            [
                "touch",
                commit_fpath,
            ],
        )
        with open(commit_fpath, "w") as f:
            f.write(current_hash)

        util.run(
            [
                "rm",
                "-rf",
                "sktree/_lib/sklearn",
            ]
        )

        util.run(
            [
                "mv",
                "sktree/_lib/sklearn_fork/sklearn",
                "sktree/_lib/sklearn",
            ]
        )


@click.command()
@click.option("-j", "--jobs", help="Number of parallel tasks to launch", type=int)
@click.option("--clean", is_flag=True, help="Clean build directory before build")
@click.option("--forcesubmodule", is_flag=True, help="Force submodule pull.")
@click.option("-v", "--verbose", is_flag=True, help="Print all build output, even installation")
@click.argument("meson_args", nargs=-1)
@click.pass_context
def build(ctx, meson_args, jobs=None, clean=False, forcesubmodule=False, verbose=False):
    """Build scikit-tree using submodules.

    git submodule update --recursive --remote

    To update submodule wrt latest commits:

        git submodule update --init --recursive --remote
        git add -A
        git commit

    This will update the submodule, which then must be commited so that
    git knows the submodule needs to be at a certain commit hash.
    """
    ctx.invoke(setup_submodule, forcesubmodule=forcesubmodule)

    # run build as normal
    ctx.invoke(meson.build, meson_args=meson_args, jobs=jobs, clean=clean, verbose=verbose)

[adam2392]

Thank you both for your help!

Seems like there's still some friction with using Cython w/ everything, but w/ a few loc moving stuff around, seems to work!