New and old way of Python installation and compatibility

What constitutes the new way of installing the python bindings? Is it a breaking change? Is there already documentation (I guess !960 (merged)). The master branch in dumux wants to be compatible with dune 2.8 and dune master, therefore both installation methods should be supported. Is this possible?

Until now the CI did something like this (after running dunecontrol)

# Adds all Python modules found in other Dune modules to the PYTHONPATH
./dune-common/bin/dunecontrol bexec "echo -n :\$(pwd)/python >> $(pwd)/pythonpath.txt"
export PYTHONPATH=$PYTHONPATH$(cat pythonpath.txt)
rm pythonpath.txt
# Sets up the dune-py module for JIT compilation of Python binding code
./dune-common/bin/setup-dunepy.py --opts=cmake.opts install

This fails with the current dune master. (Fails to import a dune module in Python.) I guess this is expected, because the Python bindings in dumux don't implement all the changes from !960 (merged).

I needed to install python3-venv on Ubuntu otherwise the installation crashed, is that expected? (dune-2.8 with python works fine without installing python3-venv)

added doc python labels

changed the description

changed the description

changed the description

Hi Timo, the new changes introduced some overdue overhaul handling package dependencies in CMake and the python modules. Now, the python modules and its dependent Dune modules are found more reliable and even pip-installed packages can be mixed with source packages.

1. New is that an internal DUNE virtual environment is setup during the build process. You will find this virtualenv in the build directory of dune-common (e.g. dune-common/build-cmake/dune-env). You can use this virtualenv out-of-the-box in order to use the python bindings of Dune. Actually, this is the reason why python3-venv is required now.

2. The “old” way of installing the python packages into an activated venv are expected to work as before. If some activated virtualenv is found, the packages will be installed into this virtualenv. Remark that no make install_python is necessary any more. Also, setup-dunepy.py has been removed as is called automatically when loading the first module.

3. I expect that the "PYTHONPATH" approach should also work as before, but haven’t tested it. Maybe it is the reason for your missing modules. We can have a look at this, but I would recommend to use one of the approaches above.

There were some changes in the python files defining the python package of a Dune Module. These files have to be adjusted as described in !960 (merged). Note that without these changes the bindings are not compatible any more to the master.

Have you tried to adopt these changes in dumux?

Having these files adopted should be still compatible with dune 2.8. I tested to build the current dune-mmesh/master together with core modules on 2.8. This lead to a successful installation. Of course, this requires the "old" DUNE_ENABLE_PYTHONBINDINGS and the make install_python call when building the modules.

I propose that you:

• Adopt the python package files to the new format.
• Install the Dune Python Bindings into an already activated virtual environment (2nd approach). This works with both 2.8 and master.

The workflow for an installation with an activated venv is simply:

python3 -m venv my-venv
source my-venv/bin/activate

./dune-common/bin/dunecontrol —opts=config.opts all

If you are using Core 2.8, your config.opts has to include -DDUNE_ENABLE_PYTHONBINDINGS=TRUE and you also have to call

./dune-common/bin/dunecontrol —opts=config.opts make install_python						 

The workflow for an installation with an activated venv is simply:

python3 -m venv my-venv
source my-venv/bin/activate

./dune-common/bin/dunecontrol —opts=config.opts all

If you are using Core 2.8, your config.opts has to include -DDUNE_ENABLE_PYTHONBINDINGS=TRUE and you also have to call

./dune-common/bin/dunecontrol —opts=config.opts make install_python						

Thanks for info. I was also wondering, how the setup nowadays works. The instructions on https://www.dune-project.org/doc/pythonbindings/ differ. Perhaps it would be best to distinguish explicitly between 2.7, 2.8 and newer. Comments like "works the same, but..." are not really easy for newbies and the python bindings aim at dune beginners.

And also the necessary steps to adopt existing module to the new bindings should ideally be added to this page.

And also the necessary steps to adopt existing module to the new bindings should ideally be added to this page.

That is documented in the MR and linked to in the Changelog. Since this is only a temporary problem, I don't think this should be mixed up with the installation instructions (which haven't been updated yet).

@samuel.burbulla Thanks a lot for the info, that helps a lot, I'll make the changes described in !960 (merged) to the bindings then and see if I can make it work with both 2.8 and master.

@samuel.burbulla do you see a way of adding a scenario combining core 2.8 with some git repo. as part of the github worklow testing? We could perhaps do something like pip install dune-grid==2.8.0" and then clone into the master of dune-mmsh` or something? The initial download of the git repos for core would be pointless but if there is an interest in having modules compatible with 2.8 and master of core that would be a good test perhaps to add?

Versuche gerade dafür einen Test für 2.8 mit fem master hinzuzufügen.

Ich habe die Installationsseite der Python Bindings mal überarbeitet. Kannst du drüber schauen? Dune-website !397 (merged) infrastructure/dune-website!397 (merged) Brauchen wir jetzt shared libs oder nicht?

at least for dumux I need shared libs.

Good news is that I got Dumux to work with both Dune 2.8 and Dune master. It took me a while to realize that the required changes are actually less than mentioned in !960 (merged). And our CI was relying on the fact that Python modules are only installed when executing the former setup-dunepy.py but are now already installed during configure and make which caused an inconsistent setup (#279 (closed)). This is now fixed by passing around the a venv between different CI jobs. I'll close this. Thanks for your help everyone!

closed