[Python] current python package require system mpi

Problem

Currently mpi4py is included in the python requirements in dune-common. On systems with MPI there were occasional problems with running python scripts where MPI was initialized on the C++ side. Instead we have

try:
  from mpi4py import MPI
except ImportError
  pass

in dune.common.__init__.py to avoid this issue. On system without MPI pip install dune-common fails since mpi4py cannot be build.

Solution

No idea how one can have optional dependencies in setup.py

mentioned in issue infrastructure/dune-website#97 (closed)

Strictly speaking an "optional" dependency is just not a dependency. Therefore, I would remove it from the python requirements. If someone needs MPI support in Python they have to install mpi4py into the virtual env which is then a documentation problem. Just like for C++ where you need to install an MPI implementation yourself to get MPI support.

It is a bit more complicated (or perhaps not).

The reason we introduced mpi4py quite early on while writing the Python project was that without it we got quite unpredictable segfaults. The problem seemed to be on systems with mpi installed (i.e. dune modules linked to mpi). The problems went away when mpi was initialized on the Python side before using any dune modules. That is why there is the code snippet in dune.common.__init__.py given above.

Just adding a sentence into the docu like

if you encounter a unexpected segfault please try installing mpi4py on your system

didn't seem a reasonable solution. It's different from the C++ setting since there if mpi isn't present during build it will not be linked to the libraries.

So the dependency is not really optional it is more:

if cmake found mpi and added it to the cxxflags flags then we need mpi4y. but I don't know how to do that.

That said: we had this problem right at the beginning of this whole projects and I don't know if it still is a problem or not. I removed my mpi4py in my venv and tried to get something to segfault but couldn't. But then it didn't always do that in the past either.

How about something like !1044 (closed)? I don't have the overview to understand if this would be enough. But something along these lines.

That does actually look like a good solution - didn't think of that.

MPI is a hard dependency, if the C++ code uses MPI. IMHO python has to check the C++/cmake status and depending on the this, loading a dune python module should raise an exception, if mpi4py was not initialized before...

I'm not sure how to check the MPI status on the python side, alternatively we could add a paranoia check on the C++ side and expose it in the python interface.

@tkoch Your solution looks promising.

It's the packaging we have to double check. During packaging the Python-Requires from the dune.module file are put into the pyproject.toml that is uploaded to pypi. With your change this won't be the case any more. I suppose that the mpi4py requirement in DunePythonInstallPackage will lead to an installation of mpi4py again, but it might be already one step too late when pip (resp. skbuild) reaches this point.

We just can try if a packaged module is still installable with this change. But therefore, we need a test case that fails when mpi4py is missing at the point when the libraries are compiled. If we don't find this test case, I don't see any problem.

There seems to be additional mpi4py dependencies added in dunepackaging.py and pyproject.toml. There I didn't remove it thinking that then it would still be installed during packaging. If this is not the case, can you explain what these files are for then?

ah it's only added for conda in dunepackaging.py and pyproject.toml is auto-generated.

If the requirements are written into the meta data then it should be possible to extract them here https://gitlab.dune-project.org/core/dune-common/-/blob/master/bin/dunepackaging.py#L87 again, no?

The issue is that it would have to be removed there as well - the problems I ran into recently was on a Windows machine using the linux subsystem but with no mpi there. pip install dune.common failed (with it's usual weird error output) since mpi4py could not be build (missing mpi.h).

Perhaps the solution really is to check if dune was build with mpi during import dune.generator for example and if it was simply provide an error message please install mpi4py - which should then be easy to do since dune found mpi so hopefully mpi4py will as well. That was basically @christi suggestion.

A (dirty hack) option would be to grep dune-py/config.h for ENABLE_MPI or dune-py/CMakeCache.txt for something directly after build (possibly evaluate cmake -LA -N . | grep MPI_C_HEADER_DIR executed in dune-py).

I thought that's the idea behind the metadata.cmake to pass data from cmake to python.

Wrote my answer before seeing Timo's suggestion. Parsing CMAKE_METADATA_FLAGS for HAVE_MPI=1 looks promising - I just added the flag to test and HAVE_MPI=TRUE is correctly added on my system. When parsing the metadata to generate dune-py we could have assert dune.common.metadata.HAVE_MPI==FALSE or can import mpi4py with a useful error message.

sounds good

A good approach would be to a have function dune.packagemetadata.cmakeSetting(flag). That would also simplify figuring out, for example, if albertaGrid can be used or petsc. That was in the past done rather crudely with looking at CMakeCahce.txt and config.h but the new metadata mechanism could make that a lot more elegant.

mentioned in merge request !1044 (closed)

See !1054 (merged) for a possible solution.

mentioned in merge request !1054 (merged)

closed with merge request !1054 (merged)

mentioned in commit bda81e67