[Python] JIT compilation and python tests with installed modules
Hello everyone,
First, I'd like to thank everyone for Dune in general, and since this is about Python bindings, especially the guys who made it possible to JIT compile Python bindings.
I want to use Python bindings and JIT compilation in my own module. To tests this I created the repo.
I want to execute this within a Docker container with installed modules. For a clean docker container I think its better to have it installed, since otherwise it is annoying to help CMake to find every dune-module. Furthermore, sicne i want to do this in a docker container pythons venvs are not really needed, aren't they?
It works with uninstalled modules right now but with installed modules I get different kinds of the same error message, namely
1: Comparing build directories of installed dune modules with given build directories
1: build dir /usr/local/lib/cmake/dune-common for module dune-common is expected to be unique across the given metadata - found /dune/dune-common/build-cmake
1: Dune python package could not be found.
Thus, I assume it still wants to refer to the build folders. This error occurs independent, if I delete those directories for building. See also discussion below.
Thanks in advance. @robert.kloefkorn : Do you think this refactor of this issues text and title is appropriate and makes the problem more clear for others?
Initial issue text
Hello everyone,
First, I'd like to thank everyone for Dune in general, and since this is about Python bindings, especially the guys who made it possible to JIT compile Python bindings.
TL;DR
I want to create a docker container, where I can develop my own dune module. It was easy to to this for doing C++ and running the C++ ctests. I don't really now, if my problems arise due to a wrong configured docker container or wrongly configured project files. When I enabled Python bindings several errors occur when I run, e.g.
dunecontrol --only=projectname all
cd build-cmake
ctest --verbose
This yields usually several different version of the error involving [..] is expected to be unique across the given metadata. Got [..]
, when dune-py
is created.
Usually it fails with
1: Comparing build directories of installed dune modules with given build directories
1: build dir /usr/local/lib/cmake/dune-common for module dune-common is expected to be unique across the given metadata - found /dune/dune-common/build-cmake
1: Dune python package could not be found.
or later
1: builddirs = metaData.zip_across_modules("DEPS", "DEPBUILDDIRS")
1: File "/tmp/testpythonbindings/cmake-build-debug-dockerex/dune-env/lib/python3.9/site-packages/dune/packagemetadata.py", line 489, in zip_across_modules
1: raise ValueError(f"build dir {v} for module {k} is expected to be unique across the given metadata - found {result[k]}")
1: ValueError: build dir /usr/local/lib/cmake/dune-common for module dune-common is expected to be unique across the given metadata - found /dune/dune-common/build-cmake
where it used installed module that refer to some existing or non-existing build directories.
In https://gitlab.dune-project.org/alexander.mueller/testpythonbindings/-/blob/main/projectname/python/test/test1.py
in the line v = FieldVector([0, 1, 2])
.
How can i create this docker container and setup the repository correclty that this works? What is the correct way to execute the Python tests then with ctest along the C++ tests? I want the dependencies to be installed.
Long version with several tries
- I struggled for weeks to get the Python bindings to work, and I believe I misunderstood a number of things.
- I read multiple issues as well as MRs, e.g. !960 (merged),!1086 (merged), #313, #247, and #279.
- I reviewed the docker containers and the continuous integration
- I studied how it is accomplished in dumux and other places because I wanted a module with its own namespace. The Python bindings for multiple modules are implemented differently. The folder structures may differ slightly. There remain unused files, such as setup.py.in.
Under certain conditions, it was possible to execute my Python tests, but when I added a dependency to dune.module, it stopped working for unknown reasons.
I chose to create this issue, and I'm willing to create a repository of best practices for obtaining up-to-date, functional Python bindings. Specifically, which files are optional if a "dune. [submodule]" submodule is desired?
In addition, it should detail how to execute Python tests from the build directory and use its an installed module.
In addition, this includes the case where the module depends on other Dune modules that are installed (globally) or not, as well as the relationship between Python venv and run-in-dune-env
.
To pinpoint the errors precisely, I created this MWE repository at https://gitlab.dune-project.org/alexander.mueller/testpythonbindings. In building this, I was unable to replicate the errors that exist in my actual project. I cannot even reach the point where my initial errors occurred due to new errors. In any case, I believe the errors arise from my general lack of understanding.
Possibly my problems are relatively minor, but I cannot see the solution. In this case, I'm hoping someone can assist me. Then this issue can be resolved and the repository for best practices can be removed. Thank you beforehand.
The repository currently contains the following README, which I am including here for the sake of discussion.
The README file contains the following:
Test runs
First i tried to get the MWE run with the docker images from CI.
Run python tests locally in the build folder
Not installed depencencies
Inside the docker container of 2.9 the dune modules are not installed,
since DUNECI_INSTALL_STAGE
is not set to 1, see Link, aren't they?
Running the python tests of projectname fails using the following commands. (To get this working i had to set
set (CMAKE_PREFIX_PATH "/duneci/modules/dune-common/build-cmake")
in the toplevel CMakeLists.txt
)
docker container run -it --entrypoint /bin/bash registry.dune-project.org/docker/ci/dune:2.9
git clone https://gitlab.dune-project.org/alexander.mueller/testpythonbindings.git
cd testpythonbindings/
dunecontrol --only=projectname all
cd build-cmake
ctest --verbose
test1.py
and test2.py
fails with:
ValueError: Key INSTALL_PREFIX is expected to be unique across the given metadata. Got {'/usr/local', '/duneci/install'}
Full test results
duneci@e7fac7d916b9:~/testpythonbindings/build-cmake$ ctest --verbose
UpdateCTestConfiguration from :/duneci/testpythonbindings/build-cmake/DartConfiguration.tcl
Parse Config file:/duneci/testpythonbindings/build-cmake/DartConfiguration.tcl
UpdateCTestConfiguration from :/duneci/testpythonbindings/build-cmake/DartConfiguration.tcl
Parse Config file:/duneci/testpythonbindings/build-cmake/DartConfiguration.tcl
Test project /duneci/testpythonbindings/build-cmake
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 1
Start 1: pytest1
1: Test command: /duneci/testpythonbindings/build-cmake/run-in-dune-env "python" "test1.py"
1: Test timeout computed to be: 3600
1: Comparing build directories of installed dune modules with given build directories
1: DUNE-INFO: Generating dune-py module in /duneci/modules/dune-common/build-cmake/dune-env/.cache/dune-py
1: Help on package projectname:
1:
1: NAME
1: projectname
1:
1: DESCRIPTION
1: # SPDX-FileCopyrightText: Copyright © DUNE Project contributors, see file LICENSE.md in module root
1: # SPDX-License-Identifier: LicenseRef-GPL-2.0-only-with-DUNE-exception
1:
1: PACKAGE CONTENTS
1: _projectname
1:
1: FUNCTIONS
1: add(...) method of builtins.PyCapsule instance
1: add(i: int, j: int) -> int
1:
1: A function which adds two numbers
1:
1: FILE
1: /duneci/testpythonbindings/build-cmake/python/projectname/__init__.py
1:
1:
1: Traceback (most recent call last):
1: File "/duneci/modules/dune-common/build-cmake/python/dune/common/__init__.py", line 86, in FieldVector
1: return globals()[fv](values)
1: KeyError: 'FieldVector_3'
1:
1: During handling of the above exception, another exception occurred:
1:
1: Traceback (most recent call last):
1: File "/duneci/testpythonbindings/projectname/python/test/test1.py", line 17, in <module>
1: v = FieldVector([0, 1, 2])
1: File "/duneci/modules/dune-common/build-cmake/python/dune/common/__init__.py", line 90, in FieldVector
1: cls = _loadVec(includes, typeName).FieldVector
1: File "/duneci/modules/dune-common/build-cmake/python/dune/common/__init__.py", line 74, in _loadVec
1: return generator.load(
1: File "/duneci/modules/dune-common/build-cmake/python/dune/generator/generator.py", line 172, in load
1: return self.post(moduleName, source, postscript, extraCMake)
1: File "/duneci/modules/dune-common/build-cmake/python/dune/generator/generator.py", line 124, in post
1: module = builder.load(moduleName, source, self.typeName[0], extraCMake)
1: File "/duneci/modules/dune-common/build-cmake/python/dune/generator/cmakebuilder.py", line 374, in load
1: self.initialize()
1: File "/duneci/modules/dune-common/build-cmake/python/dune/generator/cmakebuilder.py", line 616, in initialize
1: super().initialize()
1: File "/duneci/modules/dune-common/build-cmake/python/dune/generator/cmakebuilder.py", line 213, in initialize
1: self.build_dunepy_from_template(self.dune_py_dir)
1: File "/duneci/modules/dune-common/build-cmake/python/dune/generator/cmakebuilder.py", line 448, in build_dunepy_from_template
1: force = Builder.generate_dunepy_from_template(dunepy_dir, force=force)
1: File "/duneci/modules/dune-common/build-cmake/python/dune/generator/cmakebuilder.py", line 121, in generate_dunepy_from_template
1: context["install_prefix"] = metaData.unique_value_across_modules("INSTALL_PREFIX")
1: File "/duneci/modules/dune-common/build-cmake/python/dune/packagemetadata.py", line 496, in unique_value_across_modules
1: raise ValueError(f"Key {key} is expected to be unique across the given metadata. Got {values}")
1: ValueError: Key INSTALL_PREFIX is expected to be unique across the given metadata. Got {'/usr/local', '/duneci/install'}
1/2 Test #1: pytest1 ..........................***Failed 0.59 sec
test 2
Start 2: pytest2
2: Test command: /duneci/testpythonbindings/build-cmake/run-in-dune-env "python" "test2.py"
2: Test timeout computed to be: 3600
2: Comparing build directories of installed dune modules with given build directories
2: DUNE-INFO: Generating dune-py module in /duneci/modules/dune-common/build-cmake/dune-env/.cache/dune-py
2: Traceback (most recent call last):
2: File "/duneci/testpythonbindings/projectname/python/test/test2.py", line 20, in <module>
2: grid = dune.grid.structuredGrid(lowerLeft,upperRight,elements)
2: File "/duneci/modules/dune-grid/build-cmake/python/dune/grid/core.py", line 13, in structuredGrid
2: return yaspGrid(domain, dimgrid=len(lower), coordinates="equidistantoffset")
2: File "/duneci/modules/dune-grid/build-cmake/python/dune/grid/_grids.py", line 253, in yaspGrid
2: constructor = equidistantOffsetCoordinates(
2: File "/duneci/modules/dune-grid/build-cmake/python/dune/grid/_grids.py", line 165, in equidistantOffsetCoordinates
2: mod = moduleYaspCoordinates(dim,ctype)
2: File "/duneci/modules/dune-grid/build-cmake/python/dune/grid/_grids.py", line 159, in moduleYaspCoordinates
2: module = builder.load(moduleName, source, "yasp coordinates dim={dim} ctype={ct}".format(ct = ctype, dim = dim))
2: File "/duneci/modules/dune-common/build-cmake/python/dune/generator/cmakebuilder.py", line 374, in load
2: self.initialize()
2: File "/duneci/modules/dune-common/build-cmake/python/dune/generator/cmakebuilder.py", line 616, in initialize
2: super().initialize()
2: File "/duneci/modules/dune-common/build-cmake/python/dune/generator/cmakebuilder.py", line 213, in initialize
2: self.build_dunepy_from_template(self.dune_py_dir)
2: File "/duneci/modules/dune-common/build-cmake/python/dune/generator/cmakebuilder.py", line 448, in build_dunepy_from_template
2: force = Builder.generate_dunepy_from_template(dunepy_dir, force=force)
2: File "/duneci/modules/dune-common/build-cmake/python/dune/generator/cmakebuilder.py", line 121, in generate_dunepy_from_template
2: context["install_prefix"] = metaData.unique_value_across_modules("INSTALL_PREFIX")
2: File "/duneci/modules/dune-common/build-cmake/python/dune/packagemetadata.py", line 496, in unique_value_across_modules
2: raise ValueError(f"Key {key} is expected to be unique across the given metadata. Got {values}")
2: ValueError: Key INSTALL_PREFIX is expected to be unique across the given metadata. Got {'/usr/local', '/duneci/install'}
2/2 Test #2: pytest2 ..........................***Failed 0.60 sec
0% tests passed, 2 tests failed out of 2
Label Time Summary:
python = 1.19 sec*proc (2 tests)
quick = 1.19 sec*proc (2 tests)
Total Test time (real) = 1.19 sec
The following tests FAILED:
1 - pytest1 (Failed)
2 - pytest2 (Failed)
Errors while running CTest
Installed
We use the same container but install the modules as similar? to https://gitlab.dune-project.org/docker/ci/-/blob/master/base-common/duneci-install-module#L118-L119.
(To get this working i had to set
set (CMAKE_PREFIX_PATH "/duneci/install/lib/cmake/dune-common")
in the toplevel CMakeLists.txt
)
and removing the modules folder afterwards yields with the following commands
docker container run -it --entrypoint /bin/bash registry.dune-project.org/docker/ci/dune:2.9
cd modules/
dunecontrol --only=dune-common make install
cd ..
rm -rf modules/
git clone https://gitlab.dune-project.org/alexander.mueller/testpythonbindings.git
cd testpythonbindings/
/duneci/install/bin/dunecontrol --only=projectname all
the following error
ERROR: The path "/duneci/modules" given in DUNE_CONTROL_PATH does not exist.
Execution of dunecontrol terminated due to errors!
I suppose this is on purpose but why is the installation performed into the folder /duneci/install/bin
and not into /usr/bin
or usr/local/bin
?
Somehow the installed dune-common
should not depend on the downloaded build folder?
But my Linux knowledge is not good enough here, maybe.
Test with homegrown MWE docker container
This tests with my own docker container, which installs only dune-common into
/usr/local/... For the container see [
DockerImages/DockerFile](https://gitlab.dune-project.org/alexander.mueller/testpythonbindings/-/blob/main/DockerImages/DockerFile) **
test1.pyand
test2.py` fails with:
docker container run -it --entrypoint /bin/bash rath3t/dunepythonbindings:installedmodules
git clone https://gitlab.dune-project.org/alexander.mueller/testpythonbindings.git
cd testpythonbindings/
dunecontrol --only=projectname all
cd build-cmake
ctest --verbose
build dir /usr/local/lib/cmake/dune-common for module dune-common is expected to be unique across the given metadata - found /dune/dune-common/build-cmake
2: Dune python package could not be found.
Full test results
root@e4f98ef9aa1e:/testpythonbindings/build-cmake# ctest --verbose
UpdateCTestConfiguration from :/testpythonbindings/build-cmake/DartConfiguration.tcl
Parse Config file:/testpythonbindings/build-cmake/DartConfiguration.tcl
UpdateCTestConfiguration from :/testpythonbindings/build-cmake/DartConfiguration.tcl
Parse Config file:/testpythonbindings/build-cmake/DartConfiguration.tcl
Test project /testpythonbindings/build-cmake
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 1
Start 1: pytest1
1: Test command: /testpythonbindings/build-cmake/run-in-dune-env "python" "test1.py"
1: Test timeout computed to be: 3600
1: Comparing build directories of installed dune modules with given build directories
1: build dir /usr/local/lib/cmake/dune-common for module dune-common is expected to be unique across the given metadata - found /dune/dune-common/build-cmake
1: Dune python package could not be found.
1/2 Test #1: pytest1 ..........................***Skipped 0.10 sec
test 2
Start 2: pytest2
2: Test command: /testpythonbindings/build-cmake/run-in-dune-env "python" "test2.py"
2: Test timeout computed to be: 3600
2: Comparing build directories of installed dune modules with given build directories
2: build dir /usr/local/lib/cmake/dune-common for module dune-common is expected to be unique across the given metadata - found /dune/dune-common/build-cmake
2: Dune python package could not be found.
2/2 Test #2: pytest2 ..........................***Skipped 0.10 sec
100% tests passed, 0 tests failed out of 2
Label Time Summary:
python = 0.20 sec*proc (2 tests)
quick = 0.20 sec*proc (2 tests)
Total Test time (real) = 0.20 sec
The following tests did not run:
1 - pytest1 (Skipped)
2 - pytest2 (Skipped)
Not removing the dune folder where dune-common is downloaded (commenting RUN rm -rf dune/
) in DockerImages/DockerFile
yields the same error.
It can be used via the docker image rath3t/dunepythonbindings: installedmodulesNotDeletedBuildDir
.
Questions
Not installed python tests
- Is this repository almost setup correctly in terms of python bindings? I got inspired by
dumux
, since I also want to have a module with its own namespace. - Why does the above tests complain with a reference to
/dune/dune-common/build-cmake
even when the module is installed and this folder is removed? - How to run the python tests correctly without installing the module projectname?
- What needs to be present and how (installed dependencies) to circumvent the errors from above?
- Is there a magic combination of running these targets?
build_tests projectname _projectname test_python build_python_tests
. I considermetadata_install_python_package_dune,install_python_package_dune,install_python
as not needed for this question, just from the naming they are used for installing the package, aren't they?
Installing the local python module to run python tests with installed module
How should the module be installed:
From dune-project.org/dev/adding_python
pip install -v --pre --log logfile --find-links file://$PWD/dist packagename==0.1
or using metadata_install_python_package_dune,install_python_package_dune,install_python
in some combination?
Can I then copy the test above test1.py
and test2.py
somewhere and then test it with python test1.py
or does this still have to happen inside the venv and I have to write
run-in-dune-env python test1.py
? But where should the file run-in-dune-env
be in this case, since all build directories are potentially deleted?