Merge of small MRs that make changes to the Python bindings

Merges

• !931 (closed) feature/store-commit-hash-in-pypi-package
• !929 (closed) bugfix/packagemetadata
• !909 (closed) feature/generator-speed-up
• !908 (closed) feature/verbose-jit
• !923 (closed) bugfix/missing-dependecies-for-pip-install-dune-common
• !920 (closed) issue/find_installed_modules
• !928 (closed) bugfix/dunepy_fix_path_to_installed_module
• !921 (closed) bugfix/use-additional-pip-params-in-wheel-building
• !931 (closed) feature/store-commit-hash-in-pypi-package

The whole MR needs some squashing before commit

added 1 commit

• 784321a7 - [bugfix]

Compare with previous version

added 1 commit

• c1502dc3 - remove some debug output

Compare with previous version

resolved all threads

added 1 commit

• f046fead - Update dependencies in pyproject.toml.

Compare with previous version

added 1 commit

• c571acce - Rename 'Loading' to 'Compiling' in builder.

Compare with previous version

added 1 commit

• 2b2fd457 - Add logger info when configuring dune-py.

Compare with previous version

changed the description

added 1 commit

• 3bd58ea0 - fix an issue caused by missing 'venv/.cache' folder

Compare with previous version

approved this merge request

Looks ok from my side. I did a quick test with the conda build.

added 1 commit

• e719350e - fix an issue with missing import of 'os'

Compare with previous version

added 1 commit

• 0c511cb9 - Remove duplicates of numpy and portalocker.

Compare with previous version

added 1 commit

• 40ecc16b - Remove duplicate wheel.

Compare with previous version

I uploaded packages to testpypi and this works:

pip install --index-url https://test.pypi.org/simple/ --extra-index-url=https://pypi.org/simple/ dune-grid==2.8.0.dev20210213

added 2 commits

• b05b0be7 - Remove duplicate wheel.
• c114a474 - Add missing sdist command.

Compare with previous version

added 3 commits

• 40ecc16b - Remove duplicate wheel.
• 5c15fba6 - revert removal of generation of wheel - this will probably cause the nightly build CI to fail
• 5b870397 - Merge branch 'features/combinedPythonMRs' of...

Compare with previous version

added 1 commit

• 35090fbe - try changing order of parameters in wheel command

Compare with previous version

I tried to compile the packages from testpypi with a conda env and that seems to fail. It's not a big show stopper though. Have not found out what the problem is.

It looked a bit more like an MPI problem, but I also figured out that my env was messed up. I'll double check.

The testpypi packages work for me after all - was some other local issue.

The testpypi packages work for me after all on my native machine. The conda problem persists but I would ignore it for now.

The PyPI mirror is now rebuilt and includes mpi4py @andreas.dedner

mentioned in merge request !922 (merged)

Now the numpy problem is back. No idea how to get around this with the wheel command as it is.

Can we get cmake to ignore the failure of building a wheel or just warn about the failure. The dune-common package was correctly installed.

The log starts with

 Error installing wheel for python package at
  /duneci/modules/dune-common/build-cmake/python
  Run
  command:/builds/infrastructure/dune-nightly-test/modules/dune-env/bin/python3;-m;pip;wheel;-w;/builds/infrastructure/dune-nightly-test/modules/share/dune/wheelhouse;--index-url=https://gitlab.dune-project.org/api/v4/projects/133/packages/pypi/simple;--no-build-isolation;-v;/duneci/modules/dune-common/build-cmake/python
  Return code: 1
  Detailed log:

correctly builds wheel for mpi4py and then

 Collecting numpy (from dune-common==2.8.0)
    1 location(s) to search for versions of numpy:
    * https://gitlab.dune-project.org/api/v4/projects/133/packages/pypi/simple/numpy/
    Getting page https://gitlab.dune-project.org/api/v4/projects/133/packages/pypi/simple/numpy/
    Looking up "https://gitlab.dune-project.org/api/v4/projects/133/packages/pypi/simple/numpy/" in the cache
    No cache entry available
    https://gitlab.dune-project.org:443 "GET /api/v4/projects/133/packages/pypi/simple/numpy/ HTTP/1.1" 200 None
    Updating cache with response from "https://gitlab.dune-project.org/api/v4/projects/133/packages/pypi/simple/numpy/"
    Caching due to etag
    Analyzing links from page https://gitlab.dune-project.org/api/v4/projects/133/packages/pypi/simple/numpy/
      Skipping link https://gitlab.dune-project.org/api/v4/projects/133/packages/pypi/files/ecb5b74c702358cdc21268ff4c37f7466357871f53a30e6f84c686952bef16a9/numpy-1.20.1-cp37-cp37m-manylinux2010_x86_64.whl#sha256=ecb5b74c702358cdc21268ff4c37f7466357871f53a30e6f84c686952bef16a9 (from https://gitlab.dune-project.org/api/v4/projects/133/packages/pypi/simple/numpy/) (requires-python:>=3.7); it is not compatible with this Python
      Skipping link https://gitlab.dune-project.org/api/v4/projects/133/packages/pypi/files/7199109fa46277be503393be9250b983f325880766f847885607d9b13848f257/numpy-1.20.1-cp38-cp38-manylinux2010_x86_64.whl#sha256=7199109fa46277be503393be9250b983f325880766f847885607d9b13848f257 (from https://gitlab.dune-project.org/api/v4/projects/133/packages/pypi/simple/numpy/) (requires-python:>=3.7); it is not compatible with this Python
      Skipping link https://gitlab.dune-project.org/api/v4/projects/133/packages/pypi/files/032be656d89bbf786d743fee11d01ef318b0781281241997558fa7950028dd29/numpy-1.20.1-cp39-cp39-manylinux2010_x86_64.whl#sha256=032be656d89bbf786d743fee11d01ef318b0781281241997558fa7950028dd29 (from https://gitlab.dune-project.org/api/v4/projects/133/packages/pypi/simple/numpy/) (requires-python:>=3.7); it is not compatible with this Python
      Skipping link https://gitlab.dune-project.org/api/v4/projects/133/packages/pypi/files/1abc02e30e3efd81a4571e00f8e62bf42e343c76698e0a3e11d9c2b3ee0d77a7/numpy-1.20.0-cp37-cp37m-manylinux2010_x86_64.whl#sha256=1abc02e30e3efd81a4571e00f8e62bf42e343c76698e0a3e11d9c2b3ee0d77a7 (from https://gitlab.dune-project.org/api/v4/projects/133/packages/pypi/simple/numpy/) (requires-python:>=3.7); it is not compatible with this Python
      Skipping link https://gitlab.dune-project.org/api/v4/projects/133/packages/pypi/files/b66a6c15d793eda7cdad986e737775aa31b9306d588c14dd0277d2dda5546150/numpy-1.20.0-cp38-cp38-manylinux2010_x86_64.whl#sha256=b66a6c15d793eda7cdad986e737775aa31b9306d588c14dd0277d2dda5546150 (from https://gitlab.dune-project.org/api/v4/projects/133/packages/pypi/simple/numpy/) (requires-python:>=3.7); it is not compatible with this Python
      Skipping link https://gitlab.dune-project.org/api/v4/projects/133/packages/pypi/files/93c2abea7bb69f47029b84ceac30ab46dfcfdb99b671ad850a333ff794a765e4/numpy-1.20.0-cp39-cp39-manylinux2010_x86_64.whl#sha256=93c2abea7bb69f47029b84ceac30ab46dfcfdb99b671ad850a333ff794a765e4 (from https://gitlab.dune-project.org/api/v4/projects/133/packages/pypi/simple/numpy/) (requires-python:>=3.7); it is not compatible with this Python
      Skipping link https://gitlab.dune-project.org/api/v4/projects/133/packages/pypi/files/a4646724fba402aa7504cd48b4b50e783296b5e10a524c7a6da62e4a8ac9698d/numpy-1.19.5-cp36-cp36m-manylinux2010_x86_64.whl#sha256=a4646724fba402aa7504cd48b4b50e783296b5e10a524c7a6da62e4a8ac9698d (from https://gitlab.dune-project.org/api/v4/projects/133/packages/pypi/simple/numpy/) (requires-python:>=3.6); it is not compatible with this Python
    Could not find a version that satisfies the requirement numpy (from dune-common==2.8.0) (from versions: )

The annoying thing of course is that numpy is available in the system that's why the rest goes through without issues. https://gitlab.dune-project.org/infrastructure/dune-nightly-test/-/jobs/210432

You should be able to remove the ERROR_MESSAGE parameter from https://gitlab.dune-project.org/core/dune-common/-/blob/master/cmake/modules/DunePythonInstallPackage.cmake#L149 which should disable the error checking in https://gitlab.dune-project.org/core/dune-common/-/blob/master/cmake/modules/DuneExecuteProcess.cmake#L39

Btw, the fix I have in mind for the PyPI mirror creation will download all wheels from PyPI regardless of platform, python version, abi etc., so it should solve this problem as well, because apparently it finds a lot of wheels in the mirror, but not the correct one.

Sounds good - I'll wait until next week then with fixing the wheel issue and see if the new approach solves it.

added 1 commit

• 23f5298d - [testing] removed error from wheel generation to check that nightly build passes

Compare with previous version

mentioned in merge request !931 (closed)

mentioned in merge request !928 (closed)

mentioned in merge request !923 (closed)

mentioned in merge request !920 (closed)

mentioned in merge request !909 (closed)

mentioned in merge request !908 (closed)

mentioned in merge request !929 (closed)

changed the description

added 1 commit

• c76994fa - In setup-dunepy switch order of call to make install_python and generating dune-py.

Compare with previous version

added 1 commit

• 39709090 - Also import LOCK_EX in except block

Compare with previous version

Should I continue to push small fixes that I find onto this branch?

Yes please do!

added 1 commit

• 17f3062b - Match argument name against 1-tuple

Compare with previous version

added 1 commit

• cef5fce2 - Move adding of git commit hash to dunepackaging.py

Compare with previous version

I found an issue (I suppose with the locking):

When I interrupt the compilation of some module (with Ctrl+C), the next run aborts with

dune.generator.exceptions.CompileError: make: *** No rule to make target `hierarchicalgrid_2fbf00a9e3ca0098fcbe4e884ec704ee'.  Stop.

Tested with

from dune.grid import structuredGrid
grid = structuredGrid([0],[1],[10])

and a fresh dune-py module.

That's pretty annoying because I found no other solution but deleting the whole dune-py module so far.

That is not the locking but the order in which code is generated or things are added to the CMakeLists.txt file. Is this the latest version because I thought I had fixed this. Btw you don't have to delete the whole dune-py it should be enough to run python rmgenerated.py hierarchicalgrid_2fbf00a9e3ca0098fcbe4e884ec704ee But that is not a great solution of course. I'll check with my version and see why this is still an issue.

Yes, it happens with the latest version.

The rmgenerated ... works fine, thanks!

added 2 commits

• 22c4b658 - catch KeyboardInterrupt while the module is being rebuild after a new module...
• 2bd8886f - Merge branch 'features/combinedPythonMRs' of...

Compare with previous version

@Samuel Burbullamailto:samuel.burbulla@mathematik.uni-stuttgart.de could you pull and try again please

Seems to work now!