dune-common issues
https://gitlab.dune-project.org/core/dune-common/-/issues
2023-06-29T09:20:59Z
https://gitlab.dune-project.org/core/dune-common/-/issues/342
`dune-py` configure failure in `dune-istl`
2023-06-29T09:20:59Z
Andreas Dedner
`dune-py` configure failure in `dune-istl`
This was discussed in a closed MR:
https://gitlab.dune-project.org/core/dune-common/-/merge_requests/1251#note_128976
This was discussed in a closed MR:
https://gitlab.dune-project.org/core/dune-common/-/merge_requests/1251#note_128976
https://gitlab.dune-project.org/core/dune-common/-/issues/341
FTBFS with gcc-13
2023-07-12T08:54:18Z
Oliver Sander
oliver.sander@tu-dresden.de
FTBFS with gcc-13
Reported by Debian for the 2.9.0 release: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1037630
```
cd /<<PKGBUILDDIR>>/build/dune/common/simd/test && /usr/bin/c++ -DENABLE_MPI=1 -DENABLE_QUADMATH=1 -DHAVE_CONFIG_H -D_GLIBCXX_USE_FL...
Reported by Debian for the 2.9.0 release: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1037630
```
cd /<<PKGBUILDDIR>>/build/dune/common/simd/test && /usr/bin/c++ -DENABLE_MPI=1 -DENABLE_QUADMATH=1 -DHAVE_CONFIG_H -D_GLIBCXX_USE_FLOAT128 -I/<<PKGBUILDDIR>>/build -I/<<PKGBUILDDIR>> -isystem /usr/lib/x86_64-linux-gnu/openmpi/include -isystem /usr/lib/x86_64-linux-gnu/openmpi/include/openmpi -std=c++17 -g -O2 -ffile-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIE -fext-numeric-literals -MD -MT dune/common/simd/test/CMakeFiles/looptest.dir/looptest.cc.o -MF CMakeFiles/looptest.dir/looptest.cc.o.d -o CMakeFiles/looptest.dir/looptest.cc.o -c /<<PKGBUILDDIR>>/build/dune/common/simd/test/looptest.cc
In file included from /usr/include/c++/13/cassert:44,
from /<<PKGBUILDDIR>>/dune/common/simd/interface.hh:17,
from /<<PKGBUILDDIR>>/dune/common/simd/simd.hh:13,
from /<<PKGBUILDDIR>>/dune/common/simd/loop.hh:13,
from /<<PKGBUILDDIR>>/build/dune/common/simd/test/looptest.cc:12:
/<<PKGBUILDDIR>>/dune/common/simd/loop.hh: In constructor ‘Dune::LoopSIMD<T, S, A>::LoopSIMD()’:
/<<PKGBUILDDIR>>/dune/common/simd/loop.hh:70:31: error: ‘uintptr_t’ does not name a type
70 | assert(reinterpret_cast<uintptr_t>(this) % std::min(alignof(LoopSIMD<T,S,A>),alignof(std::max_align_t)) == 0);
| ^~~~~~~~~
/<<PKGBUILDDIR>>/dune/common/simd/loop.hh:14:1: note: ‘uintptr_t’ is defined in header ‘<cstdint>’; did you forget to ‘#include <cstdint>’?
13 | #include <dune/common/simd/simd.hh>
+++ |+#include <cstdint>
14 | #include <dune/common/typetraits.hh>
/<<PKGBUILDDIR>>/dune/common/simd/loop.hh: In constructor ‘Dune::LoopSIMD<T, S, A>::LoopSIMD(const Dune::LoopSIMD<T, S, OA>&)’:
/<<PKGBUILDDIR>>/dune/common/simd/loop.hh:82:31: error: ‘uintptr_t’ does not name a type
82 | assert(reinterpret_cast<uintptr_t>(this) % std::min(alignof(LoopSIMD<T,S,A>),alignof(std::max_align_t)) == 0);
| ^~~~~~~~~
/<<PKGBUILDDIR>>/dune/common/simd/loop.hh:82:31: note: ‘uintptr_t’ is defined in header ‘<cstdint>’; did you forget to ‘#include <cstdint>’?
make[5]: *** [dune/common/simd/test/CMakeFiles/looptest.dir/build.make:835: dune/common/simd/test/CMakeFiles/looptest.dir/looptest.cc.o] Error 1
make[5]: *** Waiting for unfinished jobs....
```
https://gitlab.dune-project.org/core/dune-common/-/issues/340
[Python] Former JIT compilation confuses latter
2023-06-14T17:54:19Z
Alexander Müller
[Python] Former JIT compilation confuses latter
I have a new grid implementation where I'm currently adding Python bindings.
Whereas I managed to register and compile my grid, later compilation failed, no matter what object.
The code at hand is:
```python
reader = (readeriga.json, "....
I have a new grid implementation where I'm currently adding Python bindings.
Whereas I managed to register and compile my grid, later compilation failed, no matter what object.
The code at hand is:
```python
reader = (readeriga.json, "../../iga/test/auxiliaryFiles/element.ibra")
gridView = igaGrid(reader, dimgrid=2,dimworld=2)
v = FieldVector((2,1)) # this line fails
```
The error is as follows:
```python
Traceback (most recent call last):
File "/dune/dune-common/build-cmake/python/dune/common/__init__.py", line 86, in FieldVector
return globals()[fv](values)
KeyError: 'FieldVector_2'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/dune-iga/dune/python/test/readGrid.py", line 23, in <module>
v= FieldVector((2,1))
File "/dune/dune-common/build-cmake/python/dune/common/__init__.py", line 90, in FieldVector
cls = _loadVec(includes, typeName).FieldVector
File "/dune/dune-common/build-cmake/python/dune/common/__init__.py", line 74, in _loadVec
return generator.load(
File "/dune/dune-common/build-cmake/python/dune/generator/generator.py", line 172, in load
return self.post(moduleName, source, postscript, extraCMake)
File "/dune/dune-common/build-cmake/python/dune/generator/generator.py", line 124, in post
module = builder.load(moduleName, source, self.typeName[0], extraCMake)
File "/dune/dune-common/build-cmake/python/dune/generator/cmakebuilder.py", line 388, in load
module = importlib.import_module("dune.generated." + moduleName)
File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'dune.generated.fieldvector_2371cd404c10f9fc2031e51d552c58df'
```
A similar error is created for every other object I try to compile afterward. Thus, I think there is no issue regarding the FieldVector.
I researched and debugged the issue and during first problem arises in
[cmakebuilder.py685](https://gitlab.dune-project.org/core/dune-common/-/blob/releases/2.9/python/dune/generator/cmakebuilder.py?ref_type=heads#L685).
This process fails with error code -11 (I don't know what that means, if it is important, or even if it is always reproducible).
After this the [line](https://gitlab.dune-project.org/core/dune-common/-/blob/releases/2.9/python/dune/generator/cmakebuilder.py?ref_type=heads#L388) fails, since the module as never generated.
Interestingly, if I do this by hand and mimic the make command as follows:
```bash
/dune/dune-common/build-cmake/dune-env/.cache/dune-py/python/dune/generated# make -q -f CMakeFiles/fieldvector_2371cd404c10f9fc2031e51d552c58df.dir/fieldvector_2371cd404c10f9fc2031e51d552c58df.make fieldvector_2371cd404c10f9fc2031e51d552c58df.so
```
and
`bash buildScript.sh fieldvector_2371cd404c10f9fc2031e51d552c58df`
my Python script from above does not raise the exception and
```bash
DUNE-DEBUG: Module fieldvector_2371cd404c10f9fc2031e51d552c58df not loaded
DUNE-DEBUG: Loading fieldvector_2371cd404c10f9fc2031e51d552c58df
```
works as expected.
How can this behavior be explained? I refactored my code several times but no difference.
https://gitlab.dune-project.org/core/dune-common/-/issues/339
wheel installation failed during make install
2023-12-06T08:10:28Z
Simon Praetorius
wheel installation failed during make install
When installing dune modules, you typically specify where to install everything, e.g., by `-DCMAKE_INSTALL_PREFIX=<path>`. This path seems to be ignored during some wheel installation in the python bindings setup. I get the error/warning...
When installing dune modules, you typically specify where to install everything, e.g., by `-DCMAKE_INSTALL_PREFIX=<path>`. This path seems to be ignored during some wheel installation in the python bindings setup. I get the error/warning:
```
Installing wheel for python package at /opt/sources/dune/dune-geometry/build-cmake/python/. into /usr/local/share/dune/wheelhouse...
CMake Warning at /opt/sources/dune/dune-common/cmake/modules/DuneExecuteProcess.cmake:76 (message):
wheel installation failed - ignored
Run
command:/opt/sources/dune/dune-common/build-cmake/dune-env/bin/python;-m;pip;wheel;-w;/usr/local/share/dune/wheelhouse;--no-deps;/opt/sources/dune/dune-geometry/build-cmake/python/.
Return code: 2
Output:
ERROR: Exception:
Traceback (most recent call last):
File "/opt/sources/dune/dune-common/build-cmake/dune-env/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 165, in exc_logging_wrapper
status = run_func(*args)
File "/opt/sources/dune/dune-common/build-cmake/dune-env/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
return func(self, options, args)
File "/opt/sources/dune/dune-common/build-cmake/dune-env/lib/python3.10/site-packages/pip/_internal/commands/wheel.py", line 111, in run
ensure_dir(options.wheel_dir)
File "/opt/sources/dune/dune-common/build-cmake/dune-env/lib/python3.10/site-packages/pip/_internal/utils/misc.py", line 101, in ensure_dir
os.makedirs(path)
File "/usr/lib/python3.10/os.py", line 215, in makedirs
makedirs(head, exist_ok=exist_ok)
File "/usr/lib/python3.10/os.py", line 225, in makedirs
mkdir(name, mode)
PermissionError: [Errno 13] Keine Berechtigung: '/usr/local/share/dune'
Call Stack (most recent call first):
python/cmake_install.cmake:53 (dune_execute_process)
cmake_install.cmake:95 (include)
```
The error is correct, i.e., the location `/usr/local/share/dune` is by default readonly.
- How can I specify where to install wheel? (I actually don't know what this is, but if it is required, I am fine with it)
- It would be nice to get this flag as a hint in the error message.
- Is this something required? If not, why is it installed at all?
https://gitlab.dune-project.org/core/dune-common/-/issues/338
[Python] Create Python bindings when Python package has different name to dun...
2023-09-22T14:01:44Z
Alexander Müller
[Python] Create Python bindings when Python package has different name to dune module
I'm trying and struggling currently to create working python bindings for python bindings, where the pypi package should have a different name to the dune module and to the actual python package.
I'm using all 2.9 branches
I.e. my dune...
I'm trying and struggling currently to create working python bindings for python bindings, where the pypi package should have a different name to the dune module and to the actual python package.
I'm using all 2.9 branches
I.e. my dune-module is called "ikarus" which is not available at pypi. Thus, as a workaround I want to name the pypi package `pyikarus`.
Thus I want the behaviour to be `pip install pyikarus` and in the python code `import ikarus` and my cpp folder structure should also be with `ikarus`.
I investigated, and my current setup is:
In the top-level `setup.py` I have
```python
...
metadata["name"] = "pyikarus"
setup(**metadata)
```
In `python/ikarus/setup.py.in` I have
```python
REQUIRED_PACKAGES = '${RequiredPythonModules}'.replace(';',' ').split(' ')
setup(
name="pyikarus",
description="${ProjectDescription}",
version="${ProjectVersionString}",
author="${ProjectAuthor}",
author_email="${ProjectMaintainerEmail}",
packages=find_packages(exclude=["docs/*"]),
zip_safe=0,
package_data={"": ["*.so"], "pyikarus": ["data/*.cmake"]},
install_requires=REQUIRED_PACKAGES,
include_package_data=True,
)
```
and in `python/ikarus/CMakeLists.txt`
```cmake
add_subdirectory(ikarus)
set(CXX_MAX_STANDARD 20)
set(PYPI_PACKAGE_NAME "pyikarus")
dune_python_configure_bindings(
PATH
"."
PACKAGENAME
${PYPI_PACKAGE_NAME}
CMAKE_METADATA_FLAGS
DUNE_OPTS_FILE
CXX_MAX_STANDARD
)
if(POLICY CMP0087)
cmake_policy(SET CMP0087 NEW)
endif()
```
There are several points where I have to state `ikarus` or `pyikarus`. Brute forcing some variants of these options always returns errors.
Thus, if I install my uploaded package from pypi and try to compile my code with `dune-py`, I get two different error messages.
If I pass to `dune_python_configure_bindings` above the name `pyikarus`, I get errors like
<details>
<summary>pyikarus error message</summary>
```python
---------------------------------------------------------------------------
CompileError Traceback (most recent call last)
Cell In[35], line 1
----> 1 svk = iks.StVenantKirchhoff(emodul=1000,nu=0.3)
3 svkPS = svk.asPlainStress()
File /dune/dune-common/build-cmake/dune-env/lib/python3.10/site-packages/ikarus/materials.py:33, in materialConstructorDecorator..wrapper(emodul, nu)
31 includes += ["ikarus/python/finiteElements/materials/material.hh"]
32 moduleName = func.__name__+"_" + hashIt(element_type)
---> 33 module = generator.load(
34 includes=includes,
35 typeName=element_type,
36 moduleName=moduleName
37 )
39 # dispatcher = { 'NeoHooke' : NeoHooke, 'StVenantKirchhoff' : StVenantKirchhoff}
42 return eval("module."+func.__name__+"(emodul,nu)")
File /dune/dune-common/build-cmake/python/dune/generator/generator.py:172, in SimpleGenerator.load(self, includes, typeName, moduleName, extraCMake, defines, preamble, postscript, options, bufferProtocol, dynamicAttr, baseClasses, holder, *args)
168 for nr, (tn, a, o, b, d, bc, h) in enumerate( zip(typeName, args, options, bufferProtocol, dynamicAttr, baseClasses, holder) ):
169 source += self.main(nr, includes, tn, *a, options=o,
170 bufferProtocol=b, dynamicAttr=d,
171 baseClasses=bc, holder=h)
--> 172 return self.post(moduleName, source, postscript, extraCMake)
File /dune/dune-common/build-cmake/python/dune/generator/generator.py:124, in SimpleGenerator.post(self, moduleName, source, postscript, extraCMake)
122 # make sure to reload the builder here in case it got updated
123 from . import builder
--> 124 module = builder.load(moduleName, source, self.typeName[0], extraCMake)
126 return module
File /dune/dune-common/build-cmake/python/dune/generator/cmakebuilder.py:382, in Builder.load(self, moduleName, source, pythonName, extraCMake)
380 module = sys.modules.get("dune.generated." + moduleName)
381 if module is None:
--> 382 self._buildModule( moduleName, source, pythonName, extraCMake )
384 ## TODO remove barrier here
385 comm.barrier()
File /dune/dune-common/build-cmake/python/dune/generator/cmakebuilder.py:732, in MakefileBuilder._buildModule(self, moduleName, source, pythonName, extraCMake)
729 # check return code
730 if exit_code > 0:
731 # retrieve stderr output
--> 732 raise CompileError(buffer_to_str(stderr))
CompileError: /dune/dune-common/build-cmake/dune-env/.cache/dune-py/python/dune/generated/StVenantKirchhoff_de7f96cf2f383220c197a4a670079062.cc:10:10: fatal error: ikarus/finiteElements/mechanics/materials.hh: No such file or directory
10 | #include
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
```
</details>
Thus, the underlying cmake script does maybe not include the correct subdirectories to make the headers findable.
If I pass to `dune_python_configure_bindings` above the name `ikarus` I get errors like
<details>
<summary>pyikarus error message</summary>
```python
---------------------------------------------------------------------------
CompileError Traceback (most recent call last)
Cell In[4], line 1
----> 1 svk = iks.StVenantKirchhoff(emodul=1000,nu=0.3)
3 svkPS = svk.asPlainStress()
File /dune/dune-common/build-cmake/dune-env/lib/python3.10/site-packages/ikarus/materials.py:33, in materialConstructorDecorator..wrapper(emodul, nu)
31 includes += ["ikarus/python/finiteElements/materials/material.hh"]
32 moduleName = func.__name__+"_" + hashIt(element_type)
---> 33 module = generator.load(
34 includes=includes,
35 typeName=element_type,
36 moduleName=moduleName
37 )
39 # dispatcher = { 'NeoHooke' : NeoHooke, 'StVenantKirchhoff' : StVenantKirchhoff}
42 return eval("module."+func.__name__+"(emodul,nu)")
File /dune/dune-common/build-cmake/python/dune/generator/generator.py:172, in SimpleGenerator.load(self, includes, typeName, moduleName, extraCMake, defines, preamble, postscript, options, bufferProtocol, dynamicAttr, baseClasses, holder, *args)
168 for nr, (tn, a, o, b, d, bc, h) in enumerate( zip(typeName, args, options, bufferProtocol, dynamicAttr, baseClasses, holder) ):
169 source += self.main(nr, includes, tn, *a, options=o,
170 bufferProtocol=b, dynamicAttr=d,
171 baseClasses=bc, holder=h)
--> 172 return self.post(moduleName, source, postscript, extraCMake)
File /dune/dune-common/build-cmake/python/dune/generator/generator.py:124, in SimpleGenerator.post(self, moduleName, source, postscript, extraCMake)
122 # make sure to reload the builder here in case it got updated
123 from . import builder
--> 124 module = builder.load(moduleName, source, self.typeName[0], extraCMake)
126 return module
File /dune/dune-common/build-cmake/python/dune/generator/cmakebuilder.py:374, in Builder.load(self, moduleName, source, pythonName, extraCMake)
370 def load(self, moduleName, source, pythonName, extraCMake=None):
371 # check if we need to initialize dune-py either because
372 # this is the first call to load or because an external module with metadata has been registered
373 if not self.initialized or not self.externalPythonModules == getExternalPythonModules():
--> 374 self.initialize()
376 # check whether module is already compiled and build it if necessary
377 # (only try to build module on rank 0!)
378 # TODO replace if rank with something better and remove barrier further down
379 if comm.rank == 0:
File /dune/dune-common/build-cmake/python/dune/generator/cmakebuilder.py:616, in MakefileBuilder.initialize(self)
615 def initialize(self):
--> 616 super().initialize()
617 # check that the compile script is available
618 script = os.path.join(self.generated_dir,"buildScript.sh")
File /dune/dune-common/build-cmake/python/dune/generator/cmakebuilder.py:213, in Builder.initialize(self)
211 self.cacheExternalModules()
212 # create dune-py module
--> 213 self.build_dunepy_from_template(self.dune_py_dir)
214 # create tag file so that dune-py is not rebuilt on the next build
215 open(tagfile, 'a').close()
File /dune/dune-common/build-cmake/python/dune/generator/cmakebuilder.py:472, in MakefileBuilder.build_dunepy_from_template(dunepy_dir, force)
468 if not useNinja:
469 # call base class dunepy_from_template (re-initialize)
470 force = Builder.generate_dunepy_from_template(dunepy_dir, force=True)
--> 472 Builder.callCMake(["cmake"] + defaultCMakeFlags() + ["."],
473 cwd=dunepy_dir,
474 infoTxt="Configuring dune-py with CMake (make)",
475 active=True, # print details anyway
476 )
478 # obtain bash and make command generated by cmake
479 # when ninja is used makeCmd is the ninja command
480 try:
File /dune/dune-common/build-cmake/python/dune/generator/cmakebuilder.py:263, in Builder.callCMake(cmake_args, cwd, infoTxt, verbose, active, logLevel, env)
260 if cmake.returncode > 0:
261 # retrieve stderr output
262 print(stderr,stdout)
--> 263 raise CompileError(buffer_to_str(stderr))
265 return stdout, stderr
CompileError: ---- LOCK
---- ACQUIRED
-- Generating CXX compiler script for CXXFLAGS overloading on command line
---- RELEASE
---- DONE
CMake Error at /dune/dune-common/build-cmake/dune-env/lib/cmake/ikarus/ikarus-targets.cmake:61 (set_target_properties):
The link interface of target "ikarus" contains:
autodiff::autodiff
but the target was not found. Possible reasons include:
* There is a typo in the target name.
* A find_package call is missing for an IMPORTED target.
* An ALIAS target is missing.
Call Stack (most recent call first):
/dune/dune-common/build-cmake/dune-env/lib/cmake/ikarus/ikarus-config.cmake:55 (include)
/dune/dune-common/cmake/modules/DuneModuleDependencies.cmake:170 (find_package)
/dune/dune-common/cmake/modules/DuneModuleDependencies.cmake:289 (find_dune_package)
/dune/dune-common/cmake/modules/DuneModuleDependencies.cmake:336 (dune_process_dependency_leafs)
/dune/dune-common/cmake/modules/DuneModuleDependencies.cmake:46 (dune_create_dependency_leafs)
/dune/dune-common/cmake/modules/DuneProject.cmake:90 (dune_create_dependency_tree)
CMakeLists.txt:106 (dune_project)
```
</details>
which indicates by the linking error, that in some cmake file the linking is not done correctly, thus indicating the first version with `pyikarus` is better? But anyway the headers are not findable in this first case. Changes in the files in `python/ikarus/` does not change the error messages.
Can someone please point me to the correct changes. I'm willing to create then a proper MR to enable this functionality, if this is of interest. Otherwise, if this should already work can someone correct my mistakes?
The current folder structure can be found at https://github.com/ikarus-project/ikarus/tree/feature/pythonBindingsV3.
Thanks, and any help is very much appreciated.
Alexander Müller
Alexander Müller
https://gitlab.dune-project.org/core/dune-common/-/issues/337
Libraries with NO_EXPORT option are exported as module libraries
2023-07-04T11:44:33Z
Santiago Ospina De Los Ríos
sospinar@gmail.com
Libraries with NO_EXPORT option are exported as module libraries
If a library target asks to not be exported (i.e., `dune_add_library(libname NO_EXPORT [...])`) it gets excluded from the export set of the library (e.g. [this line](https://gitlab.dune-project.org/core/dune-common/-/blob/fa23cf444d632f9...
If a library target asks to not be exported (i.e., `dune_add_library(libname NO_EXPORT [...])`) it gets excluded from the export set of the library (e.g. [this line](https://gitlab.dune-project.org/core/dune-common/-/blob/fa23cf444d632f9447292cc1a9eb07e88a2fce18/cmake/modules/DuneAddLibrary.cmake#L186)). That's the expected behavior.
However, the library _name_ is still exported in the `${${ProjecName}_LIBRARIES}` variable in the export file: set [here](https://gitlab.dune-project.org/core/dune-common/-/blob/fa23cf444d632f9447292cc1a9eb07e88a2fce18/cmake/modules/DuneAddLibrary.cmake#L192) and exported [here](https://gitlab.dune-project.org/core/dune-common/-/blob/fa23cf444d632f9447292cc1a9eb07e88a2fce18/cmake/modules/DuneProject.cmake#L199). This is not correct as the library was not exported and consumers have no idea how to resolve such a library name. In particular, this fails when the consumer wants to enable all dune packages with `dune_enable_all_packages`, `dune_target_enable_all_packages` or directly linking to the dependent libraries.
The correct behavior is incidentally fixed in !1247 by having internal and external library modules: `${${ProjecName}_INTERFACE_LIBRARIES}` is set [here](https://gitlab.dune-project.org/core/dune-common/-/merge_requests/1247/diffs#a18835a94745be4f5f1e2d86451897c5accab3d3_183_214) and exported [here](https://gitlab.dune-project.org/core/dune-common/-/merge_requests/1247/diffs#28f9da52000ccbac5f7d99b068322a2dae1d5a5d_200_204), where this variable is only set for exported module libraries.
_Edit: Link to the export set is updated to the correct line._
Simon Praetorius
Simon Praetorius
https://gitlab.dune-project.org/core/dune-common/-/issues/336
Python-related build-failure on the 2.9 branch
2023-05-04T19:58:09Z
Oliver Sander
oliver.sander@tu-dresden.de
Python-related build-failure on the 2.9 branch
I cannot build the `releases/2.9` branch on my up-to-date Debian testing machine. After a fresh clone, I get
```
~/dune-2.9> ./dune-common/bin/dunecontrol all
--- going to build dune-common dune-geometry dune-localfunctions dune-uggrid...
I cannot build the `releases/2.9` branch on my up-to-date Debian testing machine. After a fresh clone, I get
```
~/dune-2.9> ./dune-common/bin/dunecontrol all
--- going to build dune-common dune-geometry dune-localfunctions dune-uggrid dune-grid dune-istl dune-matrix-vector dune-solvers dune-typetree dune-gmsh4 dune-functions dune-fufem dune-elasticity dune-gfe ---
--- calling all for dune-common ---
--- calling vcsetup for dune-common ---
--- calling cmake for dune-common ---
cmake "/home/sander/dune-2.9/dune-common"
-- The C compiler identification is GNU 12.2.0
-- The CXX compiler identification is GNU 12.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test cxx_std_flag_17
-- Performing Test cxx_std_flag_17 - Success
-- Performing Test compiler_supports_cxx17
-- Performing Test compiler_supports_cxx17 - Success
-- Looking for std::experimental::make_array<int,int>
-- Looking for std::experimental::make_array<int,int> - found
-- Looking for std::move<std::experimental::detected_t<std::decay_t,int>>
-- Looking for std::move<std::experimental::detected_t<std::decay_t,int>> - found
-- Looking for std::identity
-- Looking for std::identity - not found
-- Performing Test DUNE_HAVE_CXX_UNEVALUATED_CONTEXT_LAMBDA
-- Performing Test DUNE_HAVE_CXX_UNEVALUATED_CONTEXT_LAMBDA - Failed
-- Found LATEX: /usr/bin/latex
-- Found LatexMk: /usr/bin/latexmk (found version "Version 4.79")
-- Could NOT find Sphinx (missing: SPHINX_EXECUTABLE)
-- Found Doxygen: /usr/bin/doxygen (found version "1.9.4") found components: doxygen dot
-- Found PkgConfig: /usr/bin/pkg-config (found version "1.8.1")
-- Performing tests for dune-common (from /home/sander/dune-2.9/dune-common/cmake/modules/DuneCommonMacros.cmake)
-- Set Minimal Debug Level to 4
-- Looking for sgemm_
-- Looking for sgemm_ - not found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Looking for dgemm_
-- Looking for dgemm_ - found
-- Found BLAS: /usr/lib/x86_64-linux-gnu/libblas.so;/usr/lib/x86_64-linux-gnu/libf77blas.so;/usr/lib/x86_64-linux-gnu/libatlas.so
-- Looking for cheev_
-- Looking for cheev_ - not found
-- Looking for cheev_
-- Looking for cheev_ - found
-- Found LAPACK: /usr/lib/x86_64-linux-gnu/liblapack.so;/usr/lib/x86_64-linux-gnu/libblas.so;/usr/lib/x86_64-linux-gnu/libf77blas.so;/usr/lib/x86_64-linux-gnu/libatlas.so
-- Looking for dsyev_
-- Looking for dsyev_ - found
-- Found GMP: /usr/lib/x86_64-linux-gnu/libgmpxx.so
-- Performing Test QuadMath_COMPILES
-- Performing Test QuadMath_COMPILES - Success
-- Found QuadMath: (Supported by compiler)
-- Found MPI_C: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so (found suitable version "3.1", minimum required is "3.0")
-- Found MPI: TRUE (found suitable version "3.1", minimum required is "3.0") found components: C
-- Could NOT find TBB (set TBB_DIR to path containing TBBConfig.cmake or set PKG_CONFIG_PATH to include the location of the tbb.pc file) (missing: PkgConfigTBB_LINK_LIBRARIES PkgConfigTBB_FOUND) (found version "")
-- Could NOT find PTScotch (missing: SCOTCH_LIBRARY SCOTCHERR_LIBRARY SCOTCH_INCLUDE_DIR)
-- Found METIS: /usr/lib/x86_64-linux-gnu/libmetis.so (found version "5.1")
-- Found METIS: /usr/lib/x86_64-linux-gnu/libmetis.so (found suitable version "5.1", minimum required is "5.0")
-- Found MPI_C: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so (found version "3.1")
-- Found MPI: TRUE (found version "3.1") found components: C
-- Found ParMETIS: /usr/lib/libparmetis.so (found suitable version "4.0", minimum required is "4.0")
-- Could NOT find Vc (missing: Vc_DIR)
-- Found Python3: /usr/bin/python3 (found version "3.11.2") found components: Interpreter Development Development.Module Development.Embed
-- Found pip_/usr/bin/python3: TRUE
-- Failed to find the python package virtualenv with interpreter /usr/bin/python3. (missing: DUNE_PYTHON_virtualenv_FOUND)
-- Found venv_/usr/bin/python3: TRUE
-- Building a virtualenv in /home/sander/dune-2.9/dune-common/build-cmake/dune-env
-- Found pip_/home/sander/dune-2.9/dune-common/build-cmake/dune-env/bin/python: TRUE
-- Using scripts from /home/sander/dune-2.9/dune-common/cmake/scripts for creating doxygen stuff.
-- using /home/sander/dune-2.9/dune-common/doc/doxygen/Doxystyle to create doxystyle file
-- using C macro definitions from /home/sander/dune-2.9/dune-common/doc/doxygen/doxygen-macros for Doxygen
-- Skipping building CMake API documentation (Sphinx was not found!)
-- Installing python package abstract requirements: jinja2 mpi4py numpy pip>=21.a portalocker setuptools>=41 wheel
-- Installing python package at /home/sander/dune-2.9/dune-common/build-cmake/python/. into Dune virtual environment
-- Generating the CMake metadata file at dune/data/dune-common.cmake
-- Not adding custom target for config.h generation
-- The following OPTIONAL packages have been found:
* LATEX
* LatexMk
* Doxygen, Class documentation generator, <www.doxygen.org>
To generate the class documentation from C++ sources
* BLAS, fast linear algebra routines
* LAPACK, fast linear algebra routines
* GMP, GNU multi-precision library, <https://gmplib.org>
* QuadMath, GCC Quad-Precision Math Library, <https://gcc.gnu.org/onlinedocs/libquadmath>
* Inkscape, converts SVG images, <www.inkscape.org>
To generate the documentation with LaTeX
* Threads, Multi-threading library
* METIS (required version >= 5.0), Serial Graph Partitioning, <http://glaros.dtc.umn.edu/gkhome/metis/metis/overview>
* MPI, Message Passing Interface library
Parallel programming on multiple processors
* ParMETIS (required version >= 4.0), Parallel Graph Partitioning, <http://glaros.dtc.umn.edu/gkhome/metis/parmetis/overview>
* Python3
-- The following OPTIONAL packages have not been found:
* Sphinx, Documentation generator, <www.sphinx-doc.org>
To generate the documentation from CMake and Python sources
* TBB, Intel's Threading Building Blocks, <https://github.com/oneapi-src/oneTBB>
* PTScotch, Sequential and Parallel Graph Partitioning, <https://gitlab.inria.fr/scotch/scotch>
* Vc, C++ Vectorization library, <https://github.com/VcDevel/Vc>
For use of SIMD instructions
-- Configuring done
CMake Warning (dev) in doc/comm/CMakeLists.txt:
Policy CMP0087 is not set: Install CODE|SCRIPT allow the use of generator
expressions. Run "cmake --help-policy CMP0087" for policy details. Use
the cmake_policy command to set the policy and suppress this warning.
This warning is for project developers. Use -Wno-dev to suppress it.
-- Generating done
-- Build files have been written to: /home/sander/dune-2.9/dune-common/build-cmake
--- calling make for dune-common ---
build directory: build-cmake
cmake --build . --
[ 0%] Building CXX object CMakeFiles/dunecommon.dir/dune/common/debugalign.cc.o
[ 20%] Building CXX object CMakeFiles/dunecommon.dir/dune/common/debugallocator.cc.o
[ 20%] Building CXX object CMakeFiles/dunecommon.dir/dune/common/exceptions.cc.o
[ 20%] Building CXX object CMakeFiles/dunecommon.dir/dune/common/fmatrixev.cc.o
[ 20%] Building CXX object CMakeFiles/dunecommon.dir/dune/common/ios_state.cc.o
[ 40%] Building CXX object CMakeFiles/dunecommon.dir/dune/common/parametertree.cc.o
[ 40%] Building CXX object CMakeFiles/dunecommon.dir/dune/common/parametertreeparser.cc.o
[ 40%] Building CXX object CMakeFiles/dunecommon.dir/dune/common/path.cc.o
[ 60%] Building CXX object CMakeFiles/dunecommon.dir/dune/common/simd/test.cc.o
[ 60%] Building CXX object CMakeFiles/dunecommon.dir/dune/common/stdstreams.cc.o
[ 60%] Building CXX object CMakeFiles/dunecommon.dir/dune/common/stdthread.cc.o
[ 80%] Linking CXX static library lib/libdunecommon.a
[ 80%] Built target dunecommon
[ 80%] Building CXX object python/dune/common/CMakeFiles/_common.dir/_common.cc.o
In file included from /home/sander/dune-2.9/dune-common/dune/python/pybind11/attr.h:13,
from /home/sander/dune-2.9/dune-common/dune/python/pybind11/pybind11.h:45,
from /home/sander/dune-2.9/dune-common/dune/python/common/typeregistry.hh:21,
from /home/sander/dune-2.9/dune-common/dune/python/common/dynmatrix.hh:15,
from /home/sander/dune-2.9/dune-common/python/dune/common/_common.cc:10:
/home/sander/dune-2.9/dune-common/dune/python/pybind11/cast.h: In function ‘std::string pybind11::detail::error_string()’:
/home/sander/dune-2.9/dune-common/dune/python/pybind11/cast.h:446:36: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
446 | " " + handle(frame->f_code->co_filename).cast<std::string>() +
|
```
https://gitlab.dune-project.org/core/dune-common/-/issues/335
Add service desk badge
2023-11-06T13:24:33Z
Santiago Ospina De Los Ríos
sospinar@gmail.com
Add service desk badge
It would be nice for outside users to inform the project about issues they encounter without the need of a GitLab account. That's exactly what the [service desk](https://gitlab.dune-project.org/help/user/project/service_desk) is for and ...
It would be nice for outside users to inform the project about issues they encounter without the need of a GitLab account. That's exactly what the [service desk](https://gitlab.dune-project.org/help/user/project/service_desk) is for and now we have a nice mail alias (dune-common at dune-project.org) to offer this service.
I propose to create a [Project Badge](https://gitlab.dune-project.org/help/user/project/badges) to inform people about the email to use:
* Name: Service Desk
* Badge: ![Service Desk](https://img.shields.io/badge/Service%20Desk-dune--common%40dune--project.org-informational)
* Badge Link: https://img.shields.io/badge/Service%20Desk-dune--common%40dune--project.org-informational
* Link: https://gitlab.dune-project.org/core/dune-common/-/issues/service_desk
This needs to be done by someone with owner or maintainer rights in the settings panel... (I cannot do that)
Santiago Ospina De Los Ríos
sospinar@gmail.com
Santiago Ospina De Los Ríos
sospinar@gmail.com
https://gitlab.dune-project.org/core/dune-common/-/issues/333
TransposedMatrixWrapper does not cast properly into FieldMatrix or DenseMatrix.
2023-04-21T10:10:56Z
Robert K
TransposedMatrixWrapper does not cast properly into FieldMatrix or DenseMatrix.
The recent commit https://gitlab.dune-project.org/core/dune-grid/-/commit/b2695b3c7d169348d39f20899b64a584a667aa49 breaks the compliation of the dune-spgrid python bindings because the Jacobian is returned as transposedView of the Jacobi...
The recent commit https://gitlab.dune-project.org/core/dune-grid/-/commit/b2695b3c7d169348d39f20899b64a584a667aa49 breaks the compliation of the dune-spgrid python bindings because the Jacobian is returned as transposedView of the JacobianTransposed which is not a FieldMatrix. The return transposedView does not properly cast into FieldMatrix which is the assumed return type in the Python bindings.
https://gitlab.dune-project.org/core/dune-common/-/issues/332
How to add include directories to dune targets
2024-02-23T16:12:30Z
Santiago Ospina De Los Ríos
sospinar@gmail.com
How to add include directories to dune targets
### Issue
In order to move to a target based build system, our targets need to be equipped with the necessary include directories so that we can stop relying on project-wise include commands (i.e. `target_include_directories` vs `includ...
### Issue
In order to move to a target based build system, our targets need to be equipped with the necessary include directories so that we can stop relying on project-wise include commands (i.e. `target_include_directories` vs `include_directories`). Currently, we are using `include_directories` in `dune_project()` so that all the targets in the project see the header files of all dependencies and the project itself. This needs to be removed in favor of the target based approach (i.e. `target_include_directories`). The question here is: what is the best alternative to enforce `target_include_directories` in all dune modules?
### Alternatives
_@simon.praetorius made a great summary in https://gitlab.dune-project.org/core/dune-common/-/merge_requests/1249#note_127842_
There are some ideas how to move the include_directories property from a global property into targets:
1. Manually setting the include dir to a target. Pro: it is explicit and works right now. Con: Every module needs to create a library. It is quiet a complicated cmake statement to get right. It should actually be something like
```cmake
target_include_directories(<target> [INTERFACE]
$<BUILD_INTERFACE:${PROJECT_BINARY_DIR}>
$<BUILD_INTERFACE:${PROJECT_SOURCE_DIR}>
$<INSTALL_INTERFACE:${CMAKE_INSTALL_INCLUDEDIR}>)
```
2. Extending the function `dune_project(<target> [INTERFACE]...)` by additional arguments to directly create a module library and set the include directories on this. Pro: expects only little change in the code, can be backwards compatible. Con: now `dune_project` has two functions, configuring the module and creating a library. See !944
3. Adding a helper function just for the purpose of semiautomatically configuring a module library, e.g., `dune_add_module_library(<target> [INTERFACE]...)`. Pro: clear separation of functionality `dune_project` vs. `dune_add_library` but still some automatism. Con: Not yet clear about backwards compatible implementation.
CMake Modernization
Simon Praetorius
Simon Praetorius
https://gitlab.dune-project.org/core/dune-common/-/issues/331
[Python] Additional Cmake arguments to Python generator Dune 2.9
2023-03-27T13:23:30Z
Alexander Müller
[Python] Additional Cmake arguments to Python generator Dune 2.9
I try to use JIT compilation for my module. Since my code uses c+20, the `generator.load` call fails with
```cpp
1: /dune/dune-localfefunctions/dune/localfefunctions/expressions/rebind.hh:14:26: error: ‘remove_cvref_t’ in namespace ‘s...
I try to use JIT compilation for my module. Since my code uses c+20, the `generator.load` call fails with
```cpp
1: /dune/dune-localfefunctions/dune/localfefunctions/expressions/rebind.hh:14:26: error: ‘remove_cvref_t’ in namespace ‘std’ does not name a template type
1: 14 | auto rebind(const std::remove_cvref_t<E1>& u, const std::remove_cvref_t<E2>& v,
1: | ^~~~~~~~~~~~~~
1: /dune/dune-localfefunctions/dune/localfefunctions/expressions/rebind.hh:14:21: note: ‘std::remove_cvref_t’ is only available from C++20 onwards
1: 14 | auto rebind(const std::remove_cvref_t<E1>& u, const std::remove_cvref_t<E2>& v,
```
in an included header. My generator code looks like
```python
from dune.generator.generator import SimpleGenerator
from dune.common.hashit import hashIt
def linearElasticElement(basis, element, youngsMod, nu) :
generator = SimpleGenerator("LinearElastic", "Ikarus::Python")
element_type = f"Ikarus::LinearElastic<{basis.cppTypeName}>"
extraCM=["target_compile_features( TARGET PUBLIC cxx_std_20)"] # make sure we compile with C++20 enabled
#extraCM=["set(CMAKE_CXX_STANDARD 20)"] # make sure we compile with C++20 enabled
includes = []
includes += ["ikarus/finiteElements/mechanics/linearElastic.hh"]
includes += ["ikarus/python/finiteElements/linearElastic.hh"]
moduleName = "linearElastic_" + hashIt(element_type)
module = generator.load(
includes=includes,
typeName=element_type,
moduleName=moduleName,
extraCMake=extraCM,
holder="std::shared_ptr",
)
return module.LinearElastic(basis, element, youngsMod, nu)
```
is the `extraCMake` input argument suited for this use case?
Both `extraCM` I defined there didn't change the error message.
Thanks in advance! @andreas.dedner @tkoch @robert.kloefkorn
https://gitlab.dune-project.org/core/dune-common/-/issues/330
[discussion] drop or simplify pip & venv magic during configure
2023-04-27T14:33:51Z
Christian Engwer
[discussion] drop or simplify pip & venv magic during configure
As we have seen in !1235, the automagic handling of python dependencies leads to a couple of strange problems, e.g. packages installed automatically by `cmake` are not always found.
I would like to start a discussion on how we can simpl...
As we have seen in !1235, the automagic handling of python dependencies leads to a couple of strange problems, e.g. packages installed automatically by `cmake` are not always found.
I would like to start a discussion on how we can simply this whole cmake-venv-pip integration. My provocative statement is that it leads to more problems than it solves and that a clear error message about missing dependencies would be far more helpful.
Opinions?
https://gitlab.dune-project.org/core/dune-common/-/issues/329
Build requirements are not properly documented
2023-09-23T10:46:42Z
Carsten Gräser
graeser@math.fau.de
Build requirements are not properly documented
Currently the build instructions in `README.md` and `INSTALL` are missing a lot of very important information that should be added:
* An internet connection is needed because `pip` is used to download python packages.
* When no connecti...
Currently the build instructions in `README.md` and `INSTALL` are missing a lot of very important information that should be added:
* An internet connection is needed because `pip` is used to download python packages.
* When no connection is available, one gets several warnings and error messages.
However, the build seems to terminate successfully. Furthermore CMake says that the optional package `Python3`has been found.
Either this is not a problem and the build is successful, then we should not scare users with these error messages. Or this
means that the python-bindings cannot be used than this should be properly reported to the user.
* If the internet connection is available, it is used to download executable code from third party locations that is executed later.
* Since the required python modules are not documented it's also not possible that the user provides them on his own
to avoid this dangerous behavior. Notice that even if the python packages mentioned in the error messages are installed
globally, the attempt to download them and the mentioned error messages persists.
BTW:
One can argue that downloading and executing code from third party locations is a no-go unless the user explicitly opts in.
This ticket is not about changing this behavior now. But we should at least be honest and clearly document this and tell the
users how it can be disabled.
Markus Blatt
Markus Blatt
https://gitlab.dune-project.org/core/dune-common/-/issues/328
[Python] JIT compilation and python tests with installed modules
2023-03-01T11:23:17Z
Alexander Müller
[Python] JIT compilation and python tests with installed modules
Hello everyone,
First, I'd like to thank everyone for Dune in general, and since this is about Python bindings, especially the guys who made it possible to JIT compile Python bindings.
I want to use Python bindings and JIT compilation ...
Hello everyone,
First, I'd like to thank everyone for Dune in general, and since this is about Python bindings, especially the guys who made it possible to JIT compile Python bindings.
I want to use Python bindings and JIT compilation in my own module.
To tests this I created the [repo](https://gitlab.dune-project.org/alexander.mueller/testpythonbindings).
I want to execute this within a Docker container with _installed_ modules.
For a clean docker container I think its better to have it installed, since otherwise it is annoying to help CMake to find every dune-module. Furthermore, sicne i want to do this in a docker container pythons venvs are not really needed, aren't they?
It works with uninstalled modules right now but with installed modules I get different kinds of the same error message, namely
```python
1: Comparing build directories of installed dune modules with given build directories
1: build dir /usr/local/lib/cmake/dune-common for module dune-common is expected to be unique across the given metadata - found /dune/dune-common/build-cmake
1: Dune python package could not be found.
```
Thus, I assume it still wants to refer to the build folders. This error occurs independent, if I delete those directories for building.
See also discussion below.
Thanks in advance.
@robert.kloefkorn : Do you think this refactor of this issues text and title is appropriate and makes the problem more clear for others?
<details>
<summary>Initial issue text</summary>
Hello everyone,
First, I'd like to thank everyone for Dune in general, and since this is about Python bindings, especially the guys who made it possible to JIT compile Python bindings.
## TL;DR
I want to create a docker container, where I can develop my own dune module.
It was easy to to this for doing C++ and running the C++ ctests.
I don't really now, if my problems arise due to a wrong configured docker container or wrongly configured project files.
When I enabled Python bindings several errors occur when I run, e.g.
```shell
dunecontrol --only=projectname all
cd build-cmake
ctest --verbose
```
This yields usually several different version of the error involving `[..] is expected to be unique across the given metadata. Got [..]`, when `dune-py` is created.
Usually it fails with
```python
1: Comparing build directories of installed dune modules with given build directories
1: build dir /usr/local/lib/cmake/dune-common for module dune-common is expected to be unique across the given metadata - found /dune/dune-common/build-cmake
1: Dune python package could not be found.
```
or later
```python
1: builddirs = metaData.zip_across_modules("DEPS", "DEPBUILDDIRS")
1: File "/tmp/testpythonbindings/cmake-build-debug-dockerex/dune-env/lib/python3.9/site-packages/dune/packagemetadata.py", line 489, in zip_across_modules
1: raise ValueError(f"build dir {v} for module {k} is expected to be unique across the given metadata - found {result[k]}")
1: ValueError: build dir /usr/local/lib/cmake/dune-common for module dune-common is expected to be unique across the given metadata - found /dune/dune-common/build-cmake
```
where it used installed module that refer to some existing or non-existing build directories.
In https://gitlab.dune-project.org/alexander.mueller/testpythonbindings/-/blob/main/projectname/python/test/test1.py
in the line `v = FieldVector([0, 1, 2])`.
How can i create this docker container and setup the repository correclty that this works? What is the correct way to execute the Python tests then with ctest along the C++ tests? I want the dependencies to be installed.
## Long version with several tries
- I struggled for weeks to get the Python bindings to work, and I believe I misunderstood a number of things.
- I read multiple issues as well as MRs, e.g. !960,!1086, #313, #247, and #279.
- I reviewed the [docker containers](https://gitlab.dune-project.org/docker/ci) and the [continuous integration](https://gitlab.dune-project.org/infrastructure/dune-nightly-test)
- I studied how it is accomplished in [dumux](https://git.iws.uni-stuttgart.de/dumux-repositories/dumux) and other places because I wanted a module with its own namespace. The Python bindings for multiple modules are implemented differently. The folder structures may differ slightly. There remain unused files, such as setup.py.in.
Under certain conditions, it was possible to execute my Python tests, but when I added a dependency to dune.module, it stopped working for unknown reasons.
I chose to create this issue, and I'm willing to create a repository of best practices for obtaining up-to-date, functional Python bindings. Specifically, which files are optional if a "dune. [submodule]" submodule is desired?
In addition, it should detail how to execute Python tests from the build directory and use its an installed module.
In addition, this includes the case where the module depends on other Dune modules that are installed (globally) or not, as well as the relationship between Python venv and `run-in-dune-env`.
To pinpoint the errors precisely, I created this MWE repository at https://gitlab.dune-project.org/alexander.mueller/testpythonbindings.
In building this, I was unable to replicate the errors that exist in my actual project.
I cannot even reach the point where my initial errors occurred due to new errors.
In any case, I believe the errors arise from my general lack of understanding.
Possibly my problems are relatively minor, but I cannot see the solution. In this case, I'm hoping someone can assist me. Then this issue can be resolved and the repository for best practices can be removed. Thank you beforehand.
The repository currently contains the following README, which I am including here for the sake of discussion.
The README file contains the following:
# Test runs
First i tried to get the MWE run with the docker images from CI.
## Run python tests locally in the build folder
### Not installed depencencies
Inside the [docker container of 2.9](https://gitlab.dune-project.org/docker/ci/-/blob/master/dune-2.9/Dockerfile) the dune modules are not installed,
since `DUNECI_INSTALL_STAGE` is not set to 1, see [Link](https://gitlab.dune-project.org/docker/ci/-/blob/master/base-common/duneci-install-module#L118-L119), aren't they?
Running the python tests of projectname fails using the following commands. (To get this working i had to set
`set (CMAKE_PREFIX_PATH "/duneci/modules/dune-common/build-cmake")` in the toplevel `CMakeLists.txt`)
```shell
docker container run -it --entrypoint /bin/bash registry.dune-project.org/docker/ci/dune:2.9
git clone https://gitlab.dune-project.org/alexander.mueller/testpythonbindings.git
cd testpythonbindings/
dunecontrol --only=projectname all
cd build-cmake
ctest --verbose
```
`test1.py` and `test2.py` fails with:
```python
ValueError: Key INSTALL_PREFIX is expected to be unique across the given metadata. Got {'/usr/local', '/duneci/install'}
```
<details>
<summary>Full test results</summary>
```shell
duneci@e7fac7d916b9:~/testpythonbindings/build-cmake$ ctest --verbose
UpdateCTestConfiguration from :/duneci/testpythonbindings/build-cmake/DartConfiguration.tcl
Parse Config file:/duneci/testpythonbindings/build-cmake/DartConfiguration.tcl
UpdateCTestConfiguration from :/duneci/testpythonbindings/build-cmake/DartConfiguration.tcl
Parse Config file:/duneci/testpythonbindings/build-cmake/DartConfiguration.tcl
Test project /duneci/testpythonbindings/build-cmake
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 1
Start 1: pytest1
1: Test command: /duneci/testpythonbindings/build-cmake/run-in-dune-env "python" "test1.py"
1: Test timeout computed to be: 3600
1: Comparing build directories of installed dune modules with given build directories
1: DUNE-INFO: Generating dune-py module in /duneci/modules/dune-common/build-cmake/dune-env/.cache/dune-py
1: Help on package projectname:
1:
1: NAME
1: projectname
1:
1: DESCRIPTION
1: # SPDX-FileCopyrightText: Copyright © DUNE Project contributors, see file LICENSE.md in module root
1: # SPDX-License-Identifier: LicenseRef-GPL-2.0-only-with-DUNE-exception
1:
1: PACKAGE CONTENTS
1: _projectname
1:
1: FUNCTIONS
1: add(...) method of builtins.PyCapsule instance
1: add(i: int, j: int) -> int
1:
1: A function which adds two numbers
1:
1: FILE
1: /duneci/testpythonbindings/build-cmake/python/projectname/__init__.py
1:
1:
1: Traceback (most recent call last):
1: File "/duneci/modules/dune-common/build-cmake/python/dune/common/__init__.py", line 86, in FieldVector
1: return globals()[fv](values)
1: KeyError: 'FieldVector_3'
1:
1: During handling of the above exception, another exception occurred:
1:
1: Traceback (most recent call last):
1: File "/duneci/testpythonbindings/projectname/python/test/test1.py", line 17, in <module>
1: v = FieldVector([0, 1, 2])
1: File "/duneci/modules/dune-common/build-cmake/python/dune/common/__init__.py", line 90, in FieldVector
1: cls = _loadVec(includes, typeName).FieldVector
1: File "/duneci/modules/dune-common/build-cmake/python/dune/common/__init__.py", line 74, in _loadVec
1: return generator.load(
1: File "/duneci/modules/dune-common/build-cmake/python/dune/generator/generator.py", line 172, in load
1: return self.post(moduleName, source, postscript, extraCMake)
1: File "/duneci/modules/dune-common/build-cmake/python/dune/generator/generator.py", line 124, in post
1: module = builder.load(moduleName, source, self.typeName[0], extraCMake)
1: File "/duneci/modules/dune-common/build-cmake/python/dune/generator/cmakebuilder.py", line 374, in load
1: self.initialize()
1: File "/duneci/modules/dune-common/build-cmake/python/dune/generator/cmakebuilder.py", line 616, in initialize
1: super().initialize()
1: File "/duneci/modules/dune-common/build-cmake/python/dune/generator/cmakebuilder.py", line 213, in initialize
1: self.build_dunepy_from_template(self.dune_py_dir)
1: File "/duneci/modules/dune-common/build-cmake/python/dune/generator/cmakebuilder.py", line 448, in build_dunepy_from_template
1: force = Builder.generate_dunepy_from_template(dunepy_dir, force=force)
1: File "/duneci/modules/dune-common/build-cmake/python/dune/generator/cmakebuilder.py", line 121, in generate_dunepy_from_template
1: context["install_prefix"] = metaData.unique_value_across_modules("INSTALL_PREFIX")
1: File "/duneci/modules/dune-common/build-cmake/python/dune/packagemetadata.py", line 496, in unique_value_across_modules
1: raise ValueError(f"Key {key} is expected to be unique across the given metadata. Got {values}")
1: ValueError: Key INSTALL_PREFIX is expected to be unique across the given metadata. Got {'/usr/local', '/duneci/install'}
1/2 Test #1: pytest1 ..........................***Failed 0.59 sec
test 2
Start 2: pytest2
2: Test command: /duneci/testpythonbindings/build-cmake/run-in-dune-env "python" "test2.py"
2: Test timeout computed to be: 3600
2: Comparing build directories of installed dune modules with given build directories
2: DUNE-INFO: Generating dune-py module in /duneci/modules/dune-common/build-cmake/dune-env/.cache/dune-py
2: Traceback (most recent call last):
2: File "/duneci/testpythonbindings/projectname/python/test/test2.py", line 20, in <module>
2: grid = dune.grid.structuredGrid(lowerLeft,upperRight,elements)
2: File "/duneci/modules/dune-grid/build-cmake/python/dune/grid/core.py", line 13, in structuredGrid
2: return yaspGrid(domain, dimgrid=len(lower), coordinates="equidistantoffset")
2: File "/duneci/modules/dune-grid/build-cmake/python/dune/grid/_grids.py", line 253, in yaspGrid
2: constructor = equidistantOffsetCoordinates(
2: File "/duneci/modules/dune-grid/build-cmake/python/dune/grid/_grids.py", line 165, in equidistantOffsetCoordinates
2: mod = moduleYaspCoordinates(dim,ctype)
2: File "/duneci/modules/dune-grid/build-cmake/python/dune/grid/_grids.py", line 159, in moduleYaspCoordinates
2: module = builder.load(moduleName, source, "yasp coordinates dim={dim} ctype={ct}".format(ct = ctype, dim = dim))
2: File "/duneci/modules/dune-common/build-cmake/python/dune/generator/cmakebuilder.py", line 374, in load
2: self.initialize()
2: File "/duneci/modules/dune-common/build-cmake/python/dune/generator/cmakebuilder.py", line 616, in initialize
2: super().initialize()
2: File "/duneci/modules/dune-common/build-cmake/python/dune/generator/cmakebuilder.py", line 213, in initialize
2: self.build_dunepy_from_template(self.dune_py_dir)
2: File "/duneci/modules/dune-common/build-cmake/python/dune/generator/cmakebuilder.py", line 448, in build_dunepy_from_template
2: force = Builder.generate_dunepy_from_template(dunepy_dir, force=force)
2: File "/duneci/modules/dune-common/build-cmake/python/dune/generator/cmakebuilder.py", line 121, in generate_dunepy_from_template
2: context["install_prefix"] = metaData.unique_value_across_modules("INSTALL_PREFIX")
2: File "/duneci/modules/dune-common/build-cmake/python/dune/packagemetadata.py", line 496, in unique_value_across_modules
2: raise ValueError(f"Key {key} is expected to be unique across the given metadata. Got {values}")
2: ValueError: Key INSTALL_PREFIX is expected to be unique across the given metadata. Got {'/usr/local', '/duneci/install'}
2/2 Test #2: pytest2 ..........................***Failed 0.60 sec
0% tests passed, 2 tests failed out of 2
Label Time Summary:
python = 1.19 sec*proc (2 tests)
quick = 1.19 sec*proc (2 tests)
Total Test time (real) = 1.19 sec
The following tests FAILED:
1 - pytest1 (Failed)
2 - pytest2 (Failed)
Errors while running CTest
```
</details>
### Installed
We use the same container but install the modules as similar? to https://gitlab.dune-project.org/docker/ci/-/blob/master/base-common/duneci-install-module#L118-L119.
(To get this working i had to set
`set (CMAKE_PREFIX_PATH "/duneci/install/lib/cmake/dune-common")` in the toplevel `CMakeLists.txt`)
and removing the modules folder afterwards yields with the following commands
```shell
docker container run -it --entrypoint /bin/bash registry.dune-project.org/docker/ci/dune:2.9
cd modules/
dunecontrol --only=dune-common make install
cd ..
rm -rf modules/
git clone https://gitlab.dune-project.org/alexander.mueller/testpythonbindings.git
cd testpythonbindings/
/duneci/install/bin/dunecontrol --only=projectname all
```
the following error
```
ERROR: The path "/duneci/modules" given in DUNE_CONTROL_PATH does not exist.
Execution of dunecontrol terminated due to errors!
```
I suppose this is on purpose but why is the installation performed into the folder `/duneci/install/bin`and not into `/usr/bin` or `usr/local/bin`?
Somehow the installed `dune-common` should not depend on the downloaded build folder?
But my Linux knowledge is not good enough here, maybe.
### Test with homegrown MWE docker container
This tests with my own docker container, which installs only `dune-common into `/usr/local/...`
For the container see [`DockerImages/DockerFile`](https://gitlab.dune-project.org/alexander.mueller/testpythonbindings/-/blob/main/DockerImages/DockerFile)
**`test1.py` and `test2.py` fails with:
```shell
docker container run -it --entrypoint /bin/bash rath3t/dunepythonbindings:installedmodules
git clone https://gitlab.dune-project.org/alexander.mueller/testpythonbindings.git
cd testpythonbindings/
dunecontrol --only=projectname all
cd build-cmake
ctest --verbose
```
```python
build dir /usr/local/lib/cmake/dune-common for module dune-common is expected to be unique across the given metadata - found /dune/dune-common/build-cmake
2: Dune python package could not be found.
```
<details>
<summary>Full test results</summary>
```shell
root@e4f98ef9aa1e:/testpythonbindings/build-cmake# ctest --verbose
UpdateCTestConfiguration from :/testpythonbindings/build-cmake/DartConfiguration.tcl
Parse Config file:/testpythonbindings/build-cmake/DartConfiguration.tcl
UpdateCTestConfiguration from :/testpythonbindings/build-cmake/DartConfiguration.tcl
Parse Config file:/testpythonbindings/build-cmake/DartConfiguration.tcl
Test project /testpythonbindings/build-cmake
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 1
Start 1: pytest1
1: Test command: /testpythonbindings/build-cmake/run-in-dune-env "python" "test1.py"
1: Test timeout computed to be: 3600
1: Comparing build directories of installed dune modules with given build directories
1: build dir /usr/local/lib/cmake/dune-common for module dune-common is expected to be unique across the given metadata - found /dune/dune-common/build-cmake
1: Dune python package could not be found.
1/2 Test #1: pytest1 ..........................***Skipped 0.10 sec
test 2
Start 2: pytest2
2: Test command: /testpythonbindings/build-cmake/run-in-dune-env "python" "test2.py"
2: Test timeout computed to be: 3600
2: Comparing build directories of installed dune modules with given build directories
2: build dir /usr/local/lib/cmake/dune-common for module dune-common is expected to be unique across the given metadata - found /dune/dune-common/build-cmake
2: Dune python package could not be found.
2/2 Test #2: pytest2 ..........................***Skipped 0.10 sec
100% tests passed, 0 tests failed out of 2
Label Time Summary:
python = 0.20 sec*proc (2 tests)
quick = 0.20 sec*proc (2 tests)
Total Test time (real) = 0.20 sec
The following tests did not run:
1 - pytest1 (Skipped)
2 - pytest2 (Skipped)
```
</details>
Not removing the dune folder where dune-common is downloaded (commenting `RUN rm -rf dune/`) in `DockerImages/DockerFile` yields the same error.
It can be used via the docker image `rath3t/dunepythonbindings: installedmodulesNotDeletedBuildDir`.
# Questions
## Not installed python tests
- Is this [repository](https://gitlab.dune-project.org/alexander.mueller/testpythonbindings) almost setup correctly in terms of python bindings? I got inspired by `dumux`, since I also want to have a module with its own namespace.
- Why does the above tests complain with a reference to `/dune/dune-common/build-cmake` even when the module is installed and this folder is removed?
- How to run the python tests correctly without installing the module projectname?
- What needs to be present and how (installed dependencies) to circumvent the errors from above?
- Is there a magic combination of running these targets?
```
build_tests
projectname
_projectname
test_python
build_python_tests
```.
I consider `metadata_install_python_package_dune,install_python_package_dune,install_python` as not needed for this question, just from the naming they are used for installing the package, aren't they?
## Installing the local python module to run python tests with installed module
How should the module be installed:
From [dune-project.org/dev/adding_python](https://dune-project.org/dev/adding_python/ )
`pip install -v --pre --log logfile --find-links file://$PWD/dist packagename==0.1`
or using `metadata_install_python_package_dune,install_python_package_dune,install_python` in some combination?
Can I then copy the test above `test1.py` and `test2.py` somewhere and then test it with `python test1.py` or does this still have to happen inside the venv and I have to write
`run-in-dune-env python test1.py`? But where should the file `run-in-dune-env` be in this case, since all build directories are potentially deleted?
</details>
https://gitlab.dune-project.org/core/dune-common/-/issues/327
FindPython3 not working correctly.
2023-05-24T12:15:27Z
Robert K
FindPython3 not working correctly.
I have two Python versions installed, `3.10.9` and `3.11.1`.
The default for `python` or `python3` is `3.10.9`. For cmake > 3.20 the cmake tests find the `3.11.1` version leading to import errors later on. The old backport (used when c...
I have two Python versions installed, `3.10.9` and `3.11.1`.
The default for `python` or `python3` is `3.10.9`. For cmake > 3.20 the cmake tests find the `3.11.1` version leading to import errors later on. The old backport (used when cmake < 3.20) finds the Python version correctly. Is there a way to make the new cmake test behave correctly?
Seems like I can solve the problem by manually specifying
```
-DPython3_EXECUTABLE=/usr/bin/python
```
But it's strange that this is not the first pick for the cmake test.
https://gitlab.dune-project.org/core/dune-common/-/issues/326
Standard library's detected_or does not contain value_t
2023-02-04T22:47:32Z
Christoph Grüninger
Standard library's detected_or does not contain value_t
dune-common contains an implementation of `std::experimental::detected_or` which aligns with N4502 and [cppreference.com](https://en.cppreference.com/w/cpp/experimental/is_detected).
But GCC (13.0) have a different opinion. Instead of `...
dune-common contains an implementation of `std::experimental::detected_or` which aligns with N4502 and [cppreference.com](https://en.cppreference.com/w/cpp/experimental/is_detected).
But GCC (13.0) have a different opinion. Instead of `value_t` they use some internal variable. The proper way is to use `std::experimental::is_detected`. [libcxx](https://github.com/llvm-mirror/libcxx/blob/master/include/experimental/type_traits) contains `value_t` as expected.
This is not yet a bug report, but a note that in a not too distant future compiler will stumble over this code. This might be a compiler bug, a missed update of N4502, or we should teach our users to use `is_detected` instead of `value_t`.
https://gitlab.dune-project.org/core/dune-common/-/issues/325
dune-env and pip: Can it be disabled?
2023-04-09T14:13:57Z
Robert K
dune-env and pip: Can it be disabled?
Is there a simple way to disable pip during dunecontrol runs?
This causes problems when building DUNE on parallel clusters where one has to use the compute node also building and then this compute note does not have internet access. Fo...
Is there a simple way to disable pip during dunecontrol runs?
This causes problems when building DUNE on parallel clusters where one has to use the compute node also building and then this compute note does not have internet access. For some reason, CMake refused to believe that pip was indeed installed and would try to download and install the modules again (which would fail because of no internet).
https://gitlab.dune-project.org/core/dune-common/-/issues/323
SuiteSparse: OPTIONAL_COMPONENTS not optional due to targets being used uncon...
2022-11-23T17:26:00Z
Markus Blatt
SuiteSparse: OPTIONAL_COMPONENTS not optional due to targets being used unconditionally
Since the introduction of targets this seems to be an issue as the targets are used even when undefined:
```
-- Found SuiteSparse: /usr/lib64/libsuitesparseconfig.so (found version "4.0.2") found components: CHOLMOD LDL UMFPACK missing c...
Since the introduction of targets this seems to be an issue as the targets are used even when undefined:
```
-- Found SuiteSparse: /usr/lib64/libsuitesparseconfig.so (found version "4.0.2") found components: CHOLMOD LDL UMFPACK missing components: SPQR
CMake Error at $HOME/dune-common/cmake/modules/FindSuiteSparse.cmake:245 (target_link_libraries):
Cannot specify link libraries for target "SuiteSparse::SPQR" which is not
built by this project.
Call Stack (most recent call first):
cmake/modules/DuneIstlMacros.cmake:16 (find_package)
$HOME/dune-common/cmake/modules/DuneModuleDependencies.cmake:112 (include)
$HOME/dune-common/cmake/modules/DuneProject.cmake:123 (dune_process_dependency_macros)
CMakeLists.txt:27 (dune_project)
-- Configuring incomplete, errors occurred!
```
Will hopefully provide a fix for this myself.
https://gitlab.dune-project.org/core/dune-common/-/issues/322
[Python] set defines in compiled modules
2023-02-10T15:06:34Z
Andreas Dedner
[Python] set defines in compiled modules
Problem
-------
The `SimpleGenerator.load` method takes a `defines` argument that can be used to set preprocess defines in the generator C++ code. At the moment that is added to the files before the includes are written into the files. ...
Problem
-------
The `SimpleGenerator.load` method takes a `defines` argument that can be used to set preprocess defines in the generator C++ code. At the moment that is added to the files before the includes are written into the files. See `dune-common/python/dune/generator/generator.py`
```
source += ''.join(["#define " + d + "\n" for d in defines])
source += ''.join(["#include <" + i + ">\n" for i in includes])
```
But these defines are not added to the generated python module instance so that they are not set in modules further down the lines (includes are added).
This is probably not the expected behavior.
Solution
--------
We could add a `cppDefines` property. But we should perhaps consider adding a dictionary to the modules collecting `1cpp` properties of the module needed downstream to make it easier to extend the jit module generation process.
https://gitlab.dune-project.org/core/dune-common/-/issues/321
No documentation of Dune::TestSuite generated
2022-11-16T11:29:13Z
Simon Praetorius
No documentation of Dune::TestSuite generated
I just noticed that the utility `Dune::TestSuite` and any other utility from the `dune/common/test` directory is excluded from doxygen. So, no documentation of these classes is generated. It is excluded in multiple forms (directory is on...
I just noticed that the utility `Dune::TestSuite` and any other utility from the `dune/common/test` directory is excluded from doxygen. So, no documentation of these classes is generated. It is excluded in multiple forms (directory is on the exclude list and there is another exclude pattern containing "/test/"). What is the best strategy to add these classes to doxygen? Can we have an exclude from the exclude?