dune-python migration

As decided on the Dune developer meeting, I will migrate the python build system infrastructure currently located in dune-python to dune-common. I will try to split the migration into 3 parts:

• !218 (merged) is kinda unrelated, but used in the project (for sake of simplicity, this MR builds upon that branch)
• This MR will contain all the stuff that I consider useful for all python related projects, such as
  • finding interpreter and libraries
  • checking availability of python packages
  • defining installation rules for python packages
  • run python test commands during ctest
• An upcoming merge request (!220 (merged)) will implement everything that relates to the configure-time envsetup currently used by projects that need to run python from cmake.

To grasp the changes of this MR:

• DunePythonCommonMacros is the entry-level module, where all the logic that is executed during configuration of dune-common is located
• All the cmake modules prefixed with DunePython contain code that is intended to be used by users. It is documented with Sphinx.

added 3 commits

• 48571251 - [doc] Fix some documentation glitches
• 8ca9b0e0 - [cmake] Add functions to specify constraints on the python interpreter version
• a99a003d - [cmake] Install all new files

Compare with previous version

added 2 commits

• 7760324e - Move documentation on the FORCE{2,3} variables to DunePythonRequireVersion.cmake
• e1232c5b - remove debug statements

Compare with previous version

added 1 commit

• 6c0b497f - [bugfix] Exit the version checking script if a version has been found

Compare with previous version

added 1 commit

• 2d78e897 - Add dune_python_install_package command

Compare with previous version

resolved all discussions

We had to add

set(Python_ADDITIONAL_VERSIONS 3.4)

for Python 3.4 to be found. I don't know if this is still necessary, though.

I do not think the Python_ADDITIONAL_VERSIONS is shadowed by the checking performed by find_package, but I will check.

So you're suggesting the user should set Python_ADDITIONAL_VERSIONS instead of the build system?is this a bug in the Distribution? On Debian Stretch (CMake 3.7) all Python versions are automatically looked for.

There is an inherent chicken-egg problem with the buildsystem setting Python_ADDITIONAL_VERSIONS: The check is triggered from dune-common, but other modules can pose constraints on the interpreter version. My current approach is: Find the interpreter and nag afterwards, if it was not sufficiently new. This of course relies on distributions to define reasonably new default interpreters. Then again, in the case of the system interpreter being in a virtualenv, it is the correct procedure anyway, as the user needs to actively choose the interpreter when setting up the env (so the buildsystem does not even have a choice).

@martin.nolte Oh, I just found out I completely misunderstood the purpose of Python_ADDITIONAL_VERSIONS. I always interpreted its semantics totally different... So it turns out it IS needed to have old cmake find new pythons (Why, only the cmake god knows (if existent)). I am very sorry.

This is fixed now on the branch.

(Why, only the cmake god knows (if existent)).

I am sure there is no cmake god. Just lots of devils like for all buildsystems.

added 2 commits

• a6a6d225 - Honour the 'none' option for install location
• ef80dd26 - [bugfix] Install install_python_package.cmake correctly

Compare with previous version

While working on it, I decided to omit the functionality to install standalone python scripts , as it is so hard to correctly determine the location to move them to. If somebody direly needs scripts, I recommend adding a dummy setup.py and have it be installed as a python package to have pip figure out the correct paths. Shout, if you disagree.

So if I've already have them in setup.py, I'm good? https://github.com/dune-community/dune-xt-common/blob/master/python/setup.py

In case that's true, than I'm not shouting

@rene.milk Exactly, that is the way I am promoting as clean solution.

added 3 commits

• 4b7fd590 - [bugfix] DUNE_PYTHON_IS_VIRTUALENV -> DUNE_PYTHON_SYSTEM_IS_VIRTUALENV
• 738234eb - Implement dune_python_add_test to wrap python testing for the cmake build system
• 5921fc89 - Fix some typos/cp bugs

Compare with previous version

added 1 commit

• 984645b2 - [bugfix] 'Correctly' pass list arguments to CMake script

Compare with previous version

added 1 commit

• f707196e - [doc] Document how to invoke the python interpreter in test commands

Compare with previous version

added 2 commits

• 3af49c60 - Move the application of default on DUNE_PYTHON_INSTALL_LOCATION to the common macros
• 3a8c6fda - Check for the presence of pip, if installation is requested

Compare with previous version

added 2 commits

• ddec4ca9 - [bugfix] Set a list of uptodate python versions
• 815e47c5 - Also find the python libraries from CMake

Compare with previous version

added 1 commit

• 26ded7f5 - Check that install location 'user' is not used inside virtual environments

Compare with previous version

The branch is now in a state that includes all the features I was envisioning for it. I have redesigned many interfaces (in comparison to dune-python), so it requires thorough testing although I am using similar concepts for quite a while already. Any feedback is welcome. I will now go on and implement the MR on top that does the virtualenv stuff, as I need that to do meaningful testing with my existing projects.

added 1 commit

• 0f58c7ca - Simplify some logic about whether a package was found

Compare with previous version

added 1 commit

• c804884a - Have dune_python_find_package take an alternative interpreter

Compare with previous version

mentioned in merge request !220 (merged)

There seems to be a severe issue with finding matching versions of Python interpreter and libs with the standard CMake tests. If somebody has experience, that would be a great help.

Maybe @felix.schindler has some hints from what he did in https://github.com/dune-community/dune-pybindxi ?

I just recall that detecting the right python version did not work for me for some reason. I thus use the find script provided by pybind11, which does the job in our case: https://github.com/dune-community/dune-pybindxi/blob/master/pybind11/tools/FindPythonLibsNew.cmake

Yeah same problem, same solution I was thinking about: Sacrifice the ability to cross-compile (hard to find a good usecase for python stuff anyway) for a working test.

resolved all discussions

added 1 commit

• 168ee316 - Simplify installation of python packages

Compare with previous version

added 1 commit

• d9138f7c - [doc] Add more documentation / correct after proof-reading

Compare with previous version

unmarked as a Work In Progress

I don't understand the idea behind the name wheelhouse. So it's a subdir of /usr/share/dune/, typically, and anything can go in there as long as it's python?

Wheel house is a term to refer to a directory, where wheels are installed to. Wheels seem the only viable way to support mixing of installed and non-installed Dune modules that ship python packages. It is not "anything" that goes in there, but wheels build with pip wheel.

This is available for about a month now, I will merge at the end of the week if nobody objects.

@dominic thanks for the clarification!

merged

mentioned in commit 9227460e

mentioned in commit 8321bfce