Skip to content

refactor the way python is used in dune

Andreas Dedner requested to merge feature/cmake-python-overhaul into master

My extension of Dominic's work on this:

Changes in this MR

Case 1: not running dunecontrol in a python venv:

the internal python venv is setup. If DUNE_ENABLE_PYTHONBINDINGS=ON is set the dune python packages are installed there (editable) during the configure phase. The python tests in the build-dir work automatically with the python packages in the build specific dune-env. At the moment the only way to turn off the internal venv is to set CMAKE_DISABLE_FIND_PACKAGE_Python3=TRUE which then disables all python in the builddir.

Case 2: running dunecontrol in a python venv:

More or less the same as before but the external venv is used instead of a newly created internal one.

Python bindings

  • As pointed out above the Python bindings are now automatically installed editable into either the newly created internal env or into an active external env. So no additional call to make install_python is needed. Executing that command now leads to a non-editable installation into the venv that was active during configure our into the user/system site-packages.
  • metadata is now generated which is used when building dune-py. At the moment this is used to find available dune module sources and to set some cmake flags

Changes to existing bindings for a package dune-foo

  • An important change is that the namespace package now follows PEP 420. That means there should be no file python/dune/__init__.py anymore.
  • A standard setup.py.in is provided in dune-common so probably setup.py.in can be removed. If you have a custom setup.py.in you need to use packages=find_namespace_packages(include=['dune.*']) instead of namespace_packages=['dune'] and the setup command in setup.py.in should also include the parameter include_package_data=True Finally, the line in setup.py defining pkg now need to be pkg = '${RequiredPythonModules}'.replace(';',' ').split(' ')
  • Remove the configure_file line from python/CMakeLists.txt since this is done in the python install cmake module
  • The cmake function dune_python_install_package has been extended to set up this metadata. The call should be moved from the top level CMakeLists.txt file to the location of the setup.py.in file (i.e. python in the core modules). A typical call would be
dune_python_install_package(
  PATH "."                                           # relative path containing the package
  DEPENDS _foo                                       # if applicable: generated modules (libs)
  CMAKE_METADATA_FILE dune/foo/metadata.cmake        # metadata file to generate
  CMAKE_METADATA_FLAGS CMAKE_PACKAGE_FLAGS           # if applicable: some cmake flags to store
)

The final line could for example be CMAKE_METADATA_FLAGS SUPER_LU_ROOT (in dune-common that line contains for example CMAKE_METADATA_FLAGS CMAKE_BUILD_TYPE CMAKE_CXX_COMPILER CMAKE_CXX_FLAGS.

  • if a Python module is to be added to the package, the current version of the CMakeLists.txt files included something like install(TARGETS _grid LIBRARY DESTINATION python/dune/grid) (in dune-grid). This should be replaced by
if(SKBUILD)
  install(TARGETS _grid LIBRARY DESTINATION python/dune/grid)
endif()
  • the .gitlab.yml files can be simplified for jobs testing the python bindings

All changes have been done in companion MRs in all core modules and a simple diff there with master shows the required changes.

See also

Edited by Andreas Dedner

Merge request reports