[Python] Only looking for package in namespace/folder dune

added python label

So what exactly should work and what worked previously? Was it possible to run make install_python to install a non dune namespace package - I never tried or thought about that? So the setting is that you have a dune git repository, i.e., with dune.module but would like to have the python bindings outside the dune namespace? What exactly is the error you are getting? Is it that the package is not installed at all or that it is not found for the configuration of dune-py?

The current setup is based around installing submodules into the dune namespace package. The one place were this is used is to find the module dependencies for dune-py by looking through the packages in the dune namespace and evaluating their meta-data (i.e. their module dependencies). I don't really see how to do that differently. The idea of the new concept is that one doesn't need to set DUNE_CONTROL_PATH anymore to find the dune modules. A module which is not a dune submodule would therefore not be found and duen-py wouldn't depend on it.

But the description of your problem is a rather wage so I don't know if this is the problem...

Yes it was possible to install a non dune namespace package before. Dumux is a dune module with a dune.module file but it doesn't have a dune or python/dune folder but a dumux and python/dumux folder. For the error I have to ask @samuel.burbulla who did the testing.

The current setup is based around installing submodules into the dune namespace package.

At least in dune 2.8 everything worked fine with a dumux Python namespace package. So I guess I'm asking for a mechanism of injecting downstream packages not part of the dune namespace to become part of the dunecontrol/dune.module dependency detection mechanism.

A module which is not a dune submodule would therefore not be found and duen-py wouldn't depend on it.

Well, dumux is a dune submodule, at least in C++. I guess one of the only requirements so far was that it has a dune.module file. The Python bindings are not in the dune namespace, but dependencies could be obtained the same way via dunecontrol/dune.module.

As @andreas.dedner mentioned, packagemetadata.py:135 searches explicitly for modules with the scope dune.:

    for package in pkgutil.iter_modules(dune.__path__, dune.__name__ + "."):

If a module is not found in this scope, it will not be considered as a dependency of dune-py, and therefore code generation fails.

Adding a dummy dune.dumux package helped out to make the configuration successful. I consider this as ugly, but it shows that it is basically just the recognization of the module as a "Dune" module that goes wrong.

Is there any approach to change l.135 above to something more general?

I could call something like dune_python_module_dependency(NAMESPACE_PACKAGE dumux) (maybe it would also be possible to add packages which are not namespace packages dune_python_module_dependency(MODULE foo)). I guess this can be propagated from CMake to Python then?

The setup is that dune-py is build during the first request for a Python module requiring jit compilation. At that point the user is in some subfolder which might not have anything to do with the dune source or build folders. At this point the system looks for installed dune submodules (dune python packages are installed into the venv during configuration of the module). All submodules are collected and their module dependency graph extracted. This is done by adding metadata to the python package during the build of the C++ module. This metadata should contain all dune module on which any module providing python bindings depends. These are all collected and the cmake structure for dune-py is created.

If dumux is at the top of this chain, i.e., there is no dune module that depends on dumux then there is no way in this approach to find the dumux build directory. One needs either a dune.dumux subpackage or another dune C++ module that provides (dune.pydumux) and has dumux in it's dune.module file.

The line @samuel.burbulla highlighted is in the function def extract_metadata(ignoreImportError=False): in dune-common/python/dune/packagemetadata.py and that has to work somehow, i.e., it needs to find this metadata for all dune-like modules. If those aren't submodules how is that supposed to work?

The Python bindings are not in the dune namespace, but dependencies could be obtained the same way via dunecontrol/dune.module.

Exactly that isn't so easy - without requiring setting DUNE_CONTROL_PATH. The whole stated objective of this overhaul attempt was to simplify using the python bindings and one of the aspects was to reduce the number of environment variables required or the usage of some further scripts like setup-dunepy.

I really can't think of a way around this except for dunmux to provide both a dune submodule (which could be basically empty) so that the C++ part is picked up and then a dumux package. The dune.dumux package acts like the dunux/dune.module file, i.e., makes the dependency information available for the buildsystem and I don't think this is uglier than having a dune.module file in a repository that does not have a dune subfolder.

The only question would be if it is possible to get make install_python to install two packages? An alternative would be for each dune module (i.e. repository with a dune.module file) to register a corresponding (empty) dune submodule if it doesn't explicitly specify one. The cmake stuff could probably be extended to include that.

The only question would be if it is possible to get make install_python to install two packages? An alternative would be for each dune module (i.e. repository with a dune.module file) to register a corresponding (empty) dune submodule if it doesn't explicitly specify one. The cmake stuff could probably be extended to include that.

Perhaps one way would be to always create submodules with a particular naming scheme which include all necessary meta-data to find the actual module.

So something like

dune.common.dune-module
dune.grid.dune-module
dumux.dune-module

where dune-module is just a suggestions, as it should serve a similar purpose like dune.module.

That is similar to my last suggestion although doing that for all packages might be better then having tow cases. On the other hand that is basically the same as requiring dumux to also install a dune.dumux submodule except that we don't pollute the whole dune namespace was two packages per module.

The dune.dumux package acts like the dunux/dune.module file, i.e., makes the dependency information available for the buildsystem and I don't think this is uglier than having a dune.module file in a repository that does not have a dune subfolder.

It actually adds a dummy module to the dune namespace polluting the namespace, so definitely sounds a bit uglier than having a passive dune.module file somewhere. I don't see how having a dune subfolder is related to having a dune.module file.

Would it be possible add some other dedicated namespace module that every module adds itself to? Or is there some other mechanism to add something that can be found through python? There could be maybe some config file somewhere that every module adds itself to (its path or module name). And the config file path can be obtained by importing some config module in dune or dune.common.

@christi I just understood your proposal, also sounds like a good solution. Basically making the dune.module file content part of the Python package. Similar to the dune.module approach. I have to add something to my Python module/package to identify it as a dune module and hook it into the dune-py machinery.

dumux is currently itself a namespace package and has submodule like dumux.common dumux.discretization dumux.assembly, so I guess each then of those would have to define their own "marker" module. So it would be good if alternatively the whole namespace could be marked once.

@tkoch I don't think so. If the meta data format is designed correctly, this should all be available in a single configuration.

@tkoch the metadata is not concerned with the python bindings but with the C++ stuff.

For example the metadata generated for dune-geometry for my setup is

DEPBUILDDIRS=/home/dedner/DUNE_PYTHON/dune-geometry/build-cmake;/home/dedner/DUNE_PYTHON/dune-common/build-cmake
DEPS=dune-geometry dune-common
MODULENAME=dune-geometry
INSTALL_PREFIX=/usr/local
CMAKE_FLAGS=DUNE_OPTS_FILE:=/home/dedner/DUNE_PYTHON/config.opts

This is located in dune-geometry/build-cmake/python/dune/geometry/metadata.cmake.

How many dune subpackages are installed for one C++ module is of no consequence (in dune-common we install dune.common, dune.generator, and dune.typeregistry - only dune.common stores metadata). If some additional python package (e.g. dumux) is installed is also not relevant for this - we just need to be able to find the dune build directories for all relevant dune modules to build dune-py and that is done by iterating over the installed dune submodules and if one of them has metadata we parse that.

Ok but if dune-py needs this information and dune-py is going to be put into some folder then why not put all the metadata directly there (e.g. dune-py/metadata/dune-grid.cmake) during configure instead of "hiding" it in the modules and then retrieving it again via dune Python package imports/locations?

might be possible - there are many cases to test with the internal/external venv and installing packages from pypi. Especially in the last case it is not clear to me if one can install something into '.cache' at this stage.

one problem might also be that this metadata cache needs to be cleaned up, e.g. in case you delete a module.

but I still like @christi's idea.

As I already said, polluting the dune namespace with twice as many subpackages just to have the metadata is ugly - I at least occasionally use dir(module) to see what is available and that would lead to a huge list).

Would it really be that much of an issue to add a few files to dumux? All that should be needed is python/CMakeLists.txt, ``python/dune/CMakeLists.txt, and python/dune/dumux/CMakeLists.txt, python/dune/dumux/init.py`

Could you at least try it (e.g. by copying the four files from dune-foamgrid where you just looked at them - except that __init__.py should be empty). At least it would be good to know if that solves the problem without having to make the whole cmake functions even more complicated then they already are which I would really like to avoid.

I just tried it the other way around (I added a dumux package to dune-geometry) and make correctly installed both dune.geometry and dumux into the venv.

As I already said, polluting the dune namespace with twice as many subpackages just to have the metadata is ugly - I at least occasionally use dir(module) to see what is available and that would lead to a huge list).

As an alternative...

could the check not simply iterate over all modules and search for thouse which have the meta-data defined inside?

I believe that is already done (as I said above `dune.generator' does not have metadata).

And how would that help anyway to find metadata for a package dumux which is located somewhere else in the site-package folder? I might have completely misunderstood what you are suggesting...

PS: just realized: you were suggesting to iterate through all modules in the site-package (and all their submodules)? The current concept requires actually loading these submodues which executes __init__.py I don't think that is viable and sounds quite time consuming with the current approach.

The relevant function extract_metadata in common/python/dune/packagemetadata.py is quite straightforward so if someone wants to suggest an alternative implementation in an MR I'm happy to run my test suite with it. Probably using the pgkutil package is not the right approach - some file globbing instead would possibly due to trick.

That said: I'm willing to invest extra time in testing this once I've heard a good reason why adding a dune.dumux package as suggested above is not workable.

It is possible to add an empty dune.dumux package. This works. It was the first thing I tried and shortly mentioned it in my first comment.

Obviously just the remaining question is: Do we want to (or do we have to) expect any DUNE python package to have a python namespace dune?

I agree with @andreas.dedner that iterating over all python modules should be avoided. This already disqualifies @christi's idea of having specific submodules. How would we find them?

On the other hand, changing the way dune-py finds DUNE modules seems not to be trivial. There might be a possibility, but we already put a lot of effort into this approach.

So, I think I am fine with expecting every DUNE module having a dune namespace package. Dumux is a bit an outlier in the perspective of its subfolder structure.

Just one other complication that just occurred to me: packages are mostly installed editable which means there is only some file like dune-env/lib/python3.8/site-packages/dune-alugrid.egg-link in there which contains the path to the actual python package (in this case /home/dedner/DUNE_PYTHON/dune-alugrid/build-cmake/python). So one would need to somehow take that into account if one wanted to use file system search for the metadata files.

So, I think I am fine with expecting every DUNE module having a dune namespace package. Dumux is a bit an outlier in the perspective of its subfolder structure.

I have probably 20 dune modules lying around that don't have a dune subfolder simply because this wasn't a requirement before. So from my perspective, it's not an outlier. (OPM (e.g. https://github.com/OPM/opm-common) is another large dune-based project with doesn't have dune folders, but I guess they have their own way of adding Python bindings) I get the other perspective of course. Also, these modules don't have Python bindings so far but this requirement makes it definitely more difficult to set up...

I agree with @andreas.dedner that iterating over all python modules should be avoided. This already disqualifies @christi's idea of having specific submodules. How would we find them?

We can find them by registering the module into dune-py at configure time. Registering could be e.g. adding a file dune-py/modules/dumux, or some other mechanism. Essentially each module has to be able to add itself to a search path/pattern/list-of-module-names of some sort.

To be clear, this would mean keeping the mechanism of iterating over packages but adding the possibility of adding other namespaces than dune.

The fact that you have many modules without a dune subfolder is not really relevant - how many modules do you have that have python bindings without a python folder? Btw they probably don't even need a python folder just a folder with a dune/dumux folder, i.e., pydumux/dune/dumux would do it. Each module with python bindings has to be adapted anyway so adding 4 files to those is really that much of an issue?

There is no dune-py to register anything into - since there is no dune-py before you actually start a python script and start doing some jit compilation. At that point all dune modules have to be found. So your last suggestion doesn't make any sense to me.

Did actually the whole Python overhaul discussion and branch pass by the dumux group? If so then some mechanism should be suggest from dumux on how to improve that.

We discussed this in January 2021 we even had a special meeting on this and setup a google doc to collect different user scenarios for the python bindings. The MR was opened at least half a year ago. At no point was it even mentioned that dumux has python bindings and that they followed some different pattern compared to the core modules - at least I can find that information anywhere. I've spend weeks over the last months to get this MR stable and working for all the cases discussed at the time. I'm really a bit fed up with this whole change and really have to start working on other things. I tried my best in this issue to suggest how this can be solved but apparently those suggestion are not good enough. I'm not going to look at this anymore but of course others can modify the current mechanism if they see the need.

Did actually the whole Python overhaul discussion and branch pass by the dumux group? If so then some mechanism should be suggest from dumux on how to improve that.

We haven't discussed this and Python bindings might have not even existed in Jan 2021 in dumux. I didn't know about a google doc, but that's definitely my fault for not looking into this upcoming change more. I really highly appreciate the efforts made here with making stuff simpler and with the Python bindings in general!

There is no need to address this issue urgently. I will for now just disable the dumux Python bindings with dune master in the CI until I find time to look into a solution. Ane then I can think about if I can come up with a suggestion that would make it simpler for downstream modules outside the dune namespace.

mentioned in merge request !1049 (merged)

mentioned in commit 0cac44a9

closed with merge request !1049 (merged)

[Python] Only looking for package in namespace/folder dune

Designs

Child items ...

Activity