Refactor python package installation

What does this MR do?

This MR is a refactor of the CMake function dune_python_install_package. The function has many responsibilities that conflict with the initial intention of the function but are needed to properly configure the python bindings. So in this MR we (@andreas.dedner, @samuel.burbulla, and @santiago.ospina) identify such responsibilities and split them into separate functions. That is:

• dune_python_configure_dependencies: Installs dependencies at configure time for a project with a setup.py file. This is done by extracting a requirements file and installing them explicitely.
• dune_link_dune_py: Create the metadata in the python package. This function effectively glues a python package with the dune-py project running the bindings. Additionally, it provides a setup.py if none is given (a typical use case of dune bindings).
• dune_python_configure_package: Configures dependencies (with dune_python_configure_dependencies) and installs a project having a setup.py at configure time (notice that we opted to include !1103 (closed) directly here). Package installation is done without internet since the package is available locally and dependencies have been already fulfilled.
• dune_python_configure_bindings: Combines dune_python_configure_package and dune_link_dune_py for easy set-up of the dune python bindings.
• dune_python_install_package: Legacy command for installing packages. Should be deprecated in favor of dune_python_configure_package and dune_python_configure_bindings.

What does this MR solves?

With the current state, only one python package can be configured/installed with this function, namely, the dune python bindings. This was not the initial intention of the function and conflicts with other use cases on downstream projects before Dune 2.8. So, this refactor, allows to pick the correct use-case of the python package and keep both intended uses available.

• Add CHANGELOG entry

Closes #310 (closed)

added buildsystem python labels

assigned to @andreas.dedner

changed the description

added 1 commit

• d7cdd1a3 - Move mpi4py dependency to python configuration file

Compare with previous version

added 1 commit

• e5b64df3 - Convert bool from CMake to python

Compare with previous version

mentioned in issue #310 (closed)

added 1 commit

• 6eb568b6 - Install steuptools during pip configuration

Compare with previous version

added 1 commit

• 0ba3c17f - Install steuptools during pip configuration

Compare with previous version

The nightly tests fail with this branch: For example all python tests are skipped in https://gitlab.dune-project.org/infrastructure/dune-nightly-test/-/jobs/355189 due to a installation problem with the python packages (line 193).

added 1 commit

• c8c44992 - Use relative paths on pip installation

Compare with previous version

I would suggest to introduce a new cmake function for non dune package installation. This function is already complicated enough and there are other things that some people want to have included here (!1098 (closed) !1096 (merged) !1104 (closed)) which will complicate things even further.

Is there a reason why this one function has to handle all cases?

PS: the setup.py exists functionality and the new approach to adding mpi4py looks nice and could be added.

added 1 commit

• 4817c2a3 - Split python install package into smaller functions

Compare with previous version

added 1 commit

• 22542403 - Split python install package into smaller functions

Compare with previous version

added 1 commit

• e36b3d7f - Split python install package into smaller functions

Compare with previous version

Just to make sure I understand the use case:

I assume there is a standard (only Python sources) package in the repository of mypackage with a setup.py in some folder pypack and during configuration of that module you want to callpip install . in pypack. Requirements given in setup.py would have to be obtained from pypi so need internet access. The installation would most likely be into the dune internal venv.

During installation you would want to add the package to the dune wheelhouse.

What I have no idea is how this will work with a dune package not from source. If I write pip install mypackage I don't know if that can install a second package from pypack as well - it's an isolated build and to obtain both mypackage and pypack in the outside venv I'm not sure. But perhaps having both a dune binding package and an second package that is not a use case?

A further question: if a dune repo is allowed to only provides exactly one python package, what exactly fails with the current setup? Is it only the two points you mention in #310 (closed). The first of those is easily solved as you already did, i.e., check for a setup.py. I can see the problem with the second part - for the dune packages we distinguish between the requirements and the actual package. The first are installed with internet access the second without. We could have a requirements file and use that for the first step and keep the --no-index for the second part. But this might not be the right thing I guess.

Is there any other issue with the current setup?

added 34 commits

• e36b3d7f...183ca5b6 - 26 commits from branch master
• eb1f72c7 - Allow processes to be executed on other directories
• 9ea66182 - Remove outdated doc notes
• c76dcbcc - Handle setup dependencies from the egg info
• 50f04ecb - Move mpi4py dependency to python configuration file
• 3781039a - Convert bool from CMake to python
• 908a7993 - Install steuptools during pip configuration
• 0036794d - Use relative paths on pip installation
• d19b8ce0 - Split python install package into smaller functions

Compare with previous version

added 1 commit

• fde508f1 - Split python install package into smaller functions

Compare with previous version

added 1 commit

• 4cdd6365 - Split python install package into smaller functions

Compare with previous version

added 1 commit

• f8ccbdd5 - Split simple packages from python bindings

Compare with previous version

@andreas.dedner This is what I have at the moment. I split the dune_python_install_package into a set of functions with dedicated tasks for the python package installation:

• dune_python_configure_dependencies: Installs dependencies at configure time for a project with a setup.py file.
• dune_link_dune_py: Create the metadata in the python package. This function effectively glues a python package with the dune-py project running the bindings.
• dune_python_configure_package: Configures dependencies (with dune_python_configure_dependencies) and sets the rules to build and install any project having setup.py, setup.py.in, or none, in which case a template is provided (scripts/setup.py.in).
• dune_python_configure_bindings: Combines dune_python_configure_package and dune_python_configure_package for easy set-up of the bindings.
• dune_python_install_package: Legacy command for installing packages. Should be deprecated in favor of dune_python_configure_package and dune_python_configure_bindings.

Missing:

• dune_python_configure_package creates a folder named <package-name>.egg-info in the working diretory. This specially is annoying if the python package is in the source directory.
• Check that the nightly build is working
• Document the CMake functions
• Document the CMake CACHE global variables
• Find alternative to support from setuptools import find_namespace_packages in ubuntu 18 Make python dependency failure non-fatal for python bindings.
• Support requirements.txt in dune_python_configure_dependencies

@santiago.ospina at the moment the pipelines are failing due to an issue with no such option: --no-build-isolation https://gitlab.dune-project.org/core/dune-common/-/jobs/406000#L237 line 237. Do you have an idea why that is happening?

mentioned in merge request !1103 (closed)

added 1 commit

• 819f66d9 - [try] upgrade to a suitable version of setuptools

Compare with previous version

added 1 commit

• 9920aa8c - force setuptools>=41 - solves an issue with ubuntu 18

Compare with previous version

Status: the pipline here and nightly build seem to work now. Pip installing the packages still fail.

Test: in new environment and (to be safe) removed .cache/pip:

pip install scikit-build
cd dune-common
bin/dunepackaging.py --onlysdist

and then

pip install --pre -v --log logfile --find-links file:///DUNE-COMMONDIR/dist dune-common

fails with

  [19/21] Installing Python package at /tmp/pip-install-4xmh4bh2/dune-common/_skbuild/linux-x86_64-3.8/cmake-build/python/. into Dune virtual environment ().
  FAILED: python/CMakeFiles/build_python_package_dune /tmp/pip-install-4xmh4bh2/dune-common/_skbuild/linux-x86_64-3.8/cmake-build/python/CMakeFiles/build_python_package_dune
  cd /tmp/pip-install-4xmh4bh2/dune-common/_skbuild/linux-x86_64-3.8/cmake-build/python && /tmp/dune-env/bin/python -m pip install --no-build-isolation --no-warn-script-location --no-index  --editable .
  Obtaining file:///tmp/pip-install-4xmh4bh2/dune-common/_skbuild/linux-x86_64-3.8/cmake-build/python
    Checking if build backend supports build_editable: started
    Checking if build backend supports build_editable: finished with status 'done'
    Preparing metadata (pyproject.toml): started
    Preparing metadata (pyproject.toml): finished with status 'done'
  Requirement already satisfied: wheel in /tmp/pip-build-env-054ie0hp/overlay/lib/python3.8/site-packages (from dune-common==2.9.0) (0.37.1)
  Requirement already satisfied: pip>=21.a in /tmp/pip-build-env-054ie0hp/overlay/lib/python3.8/site-packages (from dune-common==2.9.0) (22.1.2)
  ERROR: Could not find a version that satisfies the requirement numpy (from dune-common) (from versions: none)
  ERROR: No matching distribution found for numpy

added 1 commit

• 949ccb66 - add back into dune.module

Compare with previous version

I would suggest to revert the removal (and deprecation) of PythonRequires, If I understood correctly this is more a cosmetic change which is not directly related to the issues this MR is supposed to address. It just makes it more difficult to understand which changes this MR is making.

An issue is that looking at the diffs hardly helps here due to significant (and reasonable) reorganization of the code. I'm wondering which other 'minor' changes I might not have noticed.

added 1 commit

• 0a0afead - Apply suggestions

Compare with previous version

added 1 commit

• 6a9acb68 - Improve documentation and readability

Compare with previous version

added 1 commit

• 6bf9bc79 - avoid fatal error on venv configuration error

Compare with previous version

added 2 commits

• fb6435e9 - Restore pyproject.toml and setup.py.
• ec2fa099 - Apply changes of feature/pip-editable-in-configure.

Compare with previous version

added 1 commit

• 3393a742 - Apply changes of feature/pip-editable-in-configure.

Compare with previous version

resolved all threads

@santiago.ospina Ready from my side.

I added the changes from !1103 (closed) and checked the packaging action at GitHub.

added 4 commits

• 46a351d6 - Document deprecated function
• ce3adb3c - Remove unneeded build target
• 1720e45d - Clean up documentation
• 91bc368d - Return error code from python package installation

Compare with previous version

resolved all threads

changed the description

marked this merge request as ready

changed title from Draft: Resolve "dune_python_install_package can only install packages without setup.py" to Refactor python package installation

added 1 commit

• 404fc8c3 - added change log for MR 1148 (Refactor python package installation)

Compare with previous version

marked the checklist item Add CHANGELOG entry as completed

resolved all threads

added 23 commits

• 404fc8c3...757631ef - 11 commits from branch master
• 589e3304 - Allow processes to be executed on other directories
• 0d07444f - Move mpi4py dependency to python configuration file
• 3d9a7f9c - Use relative paths on pip installation
• 162b2259 - Split python install package into smaller functions
• 2fe64d70 - [try] upgrade to a suitable version of setuptools
• a8bf2a0a - add back into dune.module
• c02bb7d1 - Improve documentation and readability
• 16918949 - avoid fatal error on venv configuration error
• 03767dfa - Apply changes of feature/pip-editable-in-configure.
• c0c6615a - Document deprecated function
• 088ef61a - Return error code from python package installation
• 8ced750a - added change log for MR 1148 (Refactor python package installation)

Compare with previous version