dune-common issues
https://gitlab.dune-project.org/core/dune-common/-/issues
2021-03-24T21:32:34Z
https://gitlab.dune-project.org/core/dune-common/-/issues/34
Local DUNE_CONTROL_PATH doesn't prevent CMAKE_MODULE_PATH being populated wit...
2021-03-24T21:32:34Z
Lukas Böger
Local DUNE_CONTROL_PATH doesn't prevent CMAKE_MODULE_PATH being populated with globally installed entries
When compiling dune modules in a local directory parallel to globally installed modules, the CMAKE_MODULE_PATH can be configured such that global cmake modules have precedence over local ones, even though DUNE_CONTROL_PATH is exclusivel...
When compiling dune modules in a local directory parallel to globally installed modules, the CMAKE_MODULE_PATH can be configured such that global cmake modules have precedence over local ones, even though DUNE_CONTROL_PATH is exclusively set to local paths and the --only option passes nothing but local modules. This can lead to issues during the configuration for different local/global module versions.
Example: all core modules 2.4.1 and dune-alugrid 2.4.0 installed in /usr. Additionaly a local directory with all core-modules, dune-typetree and dune-pdelab from the current master branch. The following command
```bash
DUNE_CONTROL_PATH=$(find `pwd` -name dune.module | tr '\n' ':') ./dune-common/bin/dunecontrol --only=dune-common,dune-localfunctions,dune-pdelab,dune-geometry,dune-typetree,dune-istl,dune-grid configure
```
fails with the output
```
Call Stack (most recent call first):
cmake/modules/DunePdelabMacros.cmake:2 (include)
[pwd]/dune-common/cmake/modules/DuneMacros.cmake:570 (include)
[pwd]/dune-common/cmake/modules/DuneMacros.cmake:714
(dune_process_dependency_macros)
CMakeLists.txt:23 (dune_project)
CMake Error at /usr/share/dune/cmake/modules/Headercheck.cmake:11
(dune_common_script_dir):
Unknown CMake command "dune_common_script_dir".
Call Stack (most recent call first):
[pwd]/dune-common/cmake/modules/DuneMacros.cmake:738
(setup_headercheck)
CMakeLists.txt:23 (dune_project)
```
where obviously /usr/share/dune/cmake/modules/Headercheck.cmake has been loaded instead of the local ./dune-common/cmake/modules/Headercheck.cmake though no global directories are involved.
This mechanism can apparently also appear with global, non-DUNE components involved in the compilation, like the std library.
2017-11-01
https://gitlab.dune-project.org/core/dune-common/-/issues/362
Assert and debug levels for error checking
2024-02-22T15:36:08Z
Simon Praetorius
Assert and debug levels for error checking
### Summary
In several dune core modules, error checking is enabled differently.
- In dune-common `boundschecking.hh` a macro `DUNE_ASSERT_BOUNDS` is enabled, if the variable `DUNE_CHECK_BOUNDS` is set.
- In `densematrix.hh` and `diagon...
### Summary
In several dune core modules, error checking is enabled differently.
- In dune-common `boundschecking.hh` a macro `DUNE_ASSERT_BOUNDS` is enabled, if the variable `DUNE_CHECK_BOUNDS` is set.
- In `densematrix.hh` and `diagonalmatrix.hh` additionally, the macro `DUNE_FMatrix_WITH_CHECKING` is checked and if set, some additional conditions are tested.
- Also in dune-common but also in dune-istl, the macro `DUNE_ISTL_WITH_CHECKING` is available for enabling extra checks in parallel code and in istl containers
- Then we have the classical `assert` macro as well as the extra macro `DUNE_ASSERT_AND_RETURN`
- In `debugallocator.hh` we have another macro: `ALLOCATION_ASSERT`
- And `stdthread.hh` defines `DUNE_ASSERT_CALL_ONCE`
- The file `reservedvector.hh` additionally introduces the macro `CHECKSIZE` that is activated by `CHECK_RESERVEDVECTOR`
- In some files, explicitly `NDEBUG` is used for deactivating extra checks
This situation is not very satisfactory. Probably, I have not listed all available macros and variables to do some error checking. We need a better way!
#### Another related issue:
- The `DUNE_THROW` macro cannot be used in `constexpr` contexts. This is, because the `Dune::Exception` classes are no literal types and one cannot even construct them with a string message directly.
### How do others solve this issue?
- *wxWidgets* introduce a `wxDEBUG_LEVEL` variable (default value 1) and macros `wxASSERT` and `wxASSER_LEVEL_2` to distinguish cheap and expensive debug checks
- *Microsoft* has `ASSERT` (checks if `_DEBUG` is set), `VERIFY` (always checked), and `ASSERTE` (check with message representing the checked expression)
- *deal.ii* has `Assert`, `AssertThrow` and `AssertNothrow` macros that are enabled if `DEBUG` is set.
- *LiveV* introduces macros `ASSERT(X,A)` (deactived with `-DNODEBUG`), `ASSERT_PRE(X,A)`, `ASSERT_POST(X,A)`, `ASSERT_INV(X,A)` (activated with `-DTEST_[PRE|POST|INV]`
- *CppCoreGuidelines* introduce `Expects()` and `Ensures` macros for pre- and post-conditions. In the GSL, these are implemented using the `[[likely]]` keyword
### Proposal
- I suggest to only have a single type of macro for all checks: `DUNE_ASSERT`
- I like the design of wxWidgets to have a dedicated name for expensive asserts, `DUNE_ASSERT_LEVEL_2`. Whether it is activated or not, can be controlled by the variable `DUNE_DEBUG_LEVEL` (name of that macro is up to discussion)
- I like the design of LiveV with a message argument by default, but maybe this can be made optional.
- Additionally, one might consider to distinguish pre-/post-conditions and invariants, and may introduce also assumptions, e.g. `DUNE_ASSUME` that are always checked but might not leads to direct termination. These are infos to the compiler and the user and termination with error-messages must be enabled explicitly.
- The `DUNE_THROW` macro and exceptions should be made `constexpr`-friendly
An alternative with a flexible set of arguments to the assert macro it described in https://www.foonathan.net/2016/09/assertions/
### Related issues/merge-requests
- !1346 (Add the macro DUNE_ASSUME)
- #110 (`DUNE_THROW` fails when used within `constexpr` function)
https://gitlab.dune-project.org/core/dune-common/-/issues/361
Installation of pkg-config / cmake files depend on libdune<module>
2024-02-25T22:23:21Z
Christoph Grüninger
Installation of pkg-config / cmake files depend on libdune<module>
If a Dune module provides a library (like common, geometry, grid), pkg-config and CMake files are installed below the library path (e.g. `lib64`). If a module provides no library (like localfunctions, typetree), the same files go below a...
If a Dune module provides a library (like common, geometry, grid), pkg-config and CMake files are installed below the library path (e.g. `lib64`). If a module provides no library (like localfunctions, typetree), the same files go below a different lirary path (e.g. `lib`).
Caused by this part of our code:
https://gitlab.dune-project.org/core/dune-common/-/blob/master/cmake/modules/DuneProject.cmake#L246-L255
Is this really our intention? Why not always using one of the two variables `CMAKE_INSTALL_LIBDIR` or `DUNE_INSTALL_NONOBJECTLIBDIR`?
DUNE 2.10.0
https://gitlab.dune-project.org/core/dune-common/-/issues/360
dunecontrol does not manage installed modules correctly
2024-01-10T12:04:29Z
Simon Praetorius
dunecontrol does not manage installed modules correctly
### Summary
While most of the time people are using dune modules in the build-environment purely, there are some situations where this is not the best idea, e.g. when provided in a package management system or in a HPC cluster (e.g. wit...
### Summary
While most of the time people are using dune modules in the build-environment purely, there are some situations where this is not the best idea, e.g. when provided in a package management system or in a HPC cluster (e.g. with easybuild). The problem is, that `dunecontrol` fails to decide whether a module is an installed module or is in the build-environment in some cases:
- The pkg-config file shipped with the module is incorrect, e.g., a missing dependency or a typo (even if we do not use pkg-config for anything in dune)
- The module is installed in a path not directly next to the installation of dunecontrol. Then the `PKG_CONFIG_PATH` variable is filled incorrectly and again pkg-config does not find the module.
The underlying problem is, that the decision whether a modules is classified as "installed" or not, is just by checking whether pkg-config can find it. We want to either remove the pkg-config support in the future, or clean it up (!910). Maybe we can find a different criterion whether a module is installed or not, e.g., by installing a file in a specific place, or by adding this information into the `dune.module` file.
https://gitlab.dune-project.org/core/dune-common/-/issues/359
Deprecate parts of alignedallocator.hh and debugalign.hh that are covered by ...
2023-12-21T16:11:21Z
Christoph Grüninger
Deprecate parts of alignedallocator.hh and debugalign.hh that are covered by C++17
From reading the documentation, C++17 should provide replacements for `alignedallocator.hh` and `debugalign.hh`. `std::align` and `std::alligned_alloc` sound like suitable replacements.
We need to clarify, if we need parts of the two he...
From reading the documentation, C++17 should provide replacements for `alignedallocator.hh` and `debugalign.hh`. `std::align` and `std::alligned_alloc` sound like suitable replacements.
We need to clarify, if we need parts of the two headers, deprecate the no longer necessary part for Dune 2.10 and remove them after the release.
DUNE 2.10.0
https://gitlab.dune-project.org/core/dune-common/-/issues/358
dune-common installs Python packages and fails
2023-12-07T10:46:22Z
Yuri Vic
dune-common installs Python packages and fails
```make install``` prints this and then succeeds:
```
Installing python package
Installing wheel for python package at /usr/ports/math/dune-common/work/.build/python/. into /usr/local/share/dune/wheelhouse...
CMake Warning at /usr/ports/...
```make install``` prints this and then succeeds:
```
Installing python package
Installing wheel for python package at /usr/ports/math/dune-common/work/.build/python/. into /usr/local/share/dune/wheelhouse...
CMake Warning at /usr/ports/math/dune-common/work/dune-common-d5f5ceec68cc0dc542f175857c9bf14aa98f2c11/cmake/modules/DuneExecuteProcess.cmake:76 (message):
wheel installation failed - ignored
Run
command:/usr/ports/math/dune-common/work/.build/dune-env/bin/python;-m;pip;wheel;-w;/usr/local/share/dune/wheelhouse;/usr/ports/math/dune-common/work/.build/python/.
Return code: 2
Output:
ERROR: Exception:
Traceback (most recent call last):
File "/usr/ports/math/dune-common/work/.build/dune-env/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
status = run_func(*args)
File "/usr/ports/math/dune-common/work/.build/dune-env/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 245, in wrapper
return func(self, options, args)
File "/usr/ports/math/dune-common/work/.build/dune-env/lib/python3.9/site-packages/pip/_internal/commands/wheel.py", line 110, in run
ensure_dir(options.wheel_dir)
File "/usr/ports/math/dune-common/work/.build/dune-env/lib/python3.9/site-packages/pip/_internal/utils/misc.py", line 109, in ensure_dir
os.makedirs(path)
File "/usr/local/lib/python3.9/os.py", line 215, in makedirs
makedirs(head, exist_ok=exist_ok)
File "/usr/local/lib/python3.9/os.py", line 225, in makedirs
mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/usr/local/share/dune'
Call Stack (most recent call first):
python/cmake_install.cmake:48 (dune_execute_process)
cmake_install.cmake:113 (include)
```
It's fundamentally wrong for a C++ package to also install Python packages, because there are multiple concurrently installed Python versions, each of which needs python package to be installed individually.
If dune-common needs to install Python packages - it should create a separate project for this, for example ```dune-common-python```, that should build and install Python packages.
Version: 2.9.0
https://gitlab.dune-project.org/core/dune-common/-/issues/357
Follow-up from "[bugfix] Include config.h in lrutest.cc"
2023-12-06T13:15:29Z
Santiago Ospina De Los Ríos
sospinar@gmail.com
Follow-up from "[bugfix] Include config.h in lrutest.cc"
The following discussion from !1317 should be addressed:
- [ ] @carsten.graeser started a [discussion](https://gitlab.dune-project.org/core/dune-common/-/merge_requests/1317#note_132374): (+1 comment)
> I think one should understa...
The following discussion from !1317 should be addressed:
- [ ] @carsten.graeser started a [discussion](https://gitlab.dune-project.org/core/dune-common/-/merge_requests/1317#note_132374): (+1 comment)
> I think one should understand what's the problem here and why this was not found by the CI.
>
> Maybe related: !1262, !1310.
https://gitlab.dune-project.org/core/dune-common/-/issues/355
How are flags generated and passed to compile scripts in dune-py?
2023-11-23T20:50:18Z
Simon Praetorius
How are flags generated and passed to compile scripts in dune-py?
# Summary
I am investigate a pipeline failure
https://gitlab.dune-project.org/core/dune-common/-/jobs/531775
that shows a strange preprocessor error:
```
/builds/core/dune-common/build-cmake/dune-env/.cache/dune-py/config.h:44:34: err...
# Summary
I am investigate a pipeline failure
https://gitlab.dune-project.org/core/dune-common/-/jobs/531775
that shows a strange preprocessor error:
```
/builds/core/dune-common/build-cmake/dune-env/.cache/dune-py/config.h:44:34: error: expected value in expression
#if defined(HAVE_MPI) && HAVE_MPI
```
Actually, this would be valid code and it compiles on all compilers I have found. So, why does this happen? The reason seems to be that during the generation of the compile command/flags in dune-py the flag `HAVE_MPI=` is set without a value. I am trying the MR !1310 where mpi flags are not set via `ENABLE_MPI=1` but directly as `HAVE_MPI=1`. Thus, if no mpi is enabled, no variable is defined. This is not a problem anywhere else. So, the question is where and how this `HAVE_MPI` flag is collected and put into a list of compile-flags? And why is it set to an empty value?
https://gitlab.dune-project.org/core/dune-common/-/issues/354
Compiler warning pollution from dependencies
2023-11-06T12:57:49Z
Alexander Müller
Compiler warning pollution from dependencies
I tried for my dune module to really go hard on fix warnings from clang.
The output is polluted by giving warnings on all other included modules/header from e.g. `dune-functions,yasp_grid,dune-geometry,...`.
Now I read about the fact th...
I tried for my dune module to really go hard on fix warnings from clang.
The output is polluted by giving warnings on all other included modules/header from e.g. `dune-functions,yasp_grid,dune-geometry,...`.
Now I read about the fact that the compilers may not produce warning for system headers.
https://cmake.org/cmake/help/latest/command/target_include_directories.html
>If SYSTEM is specified, the compiler will be told the directories are meant as system include directories on some platforms. This may have effects such as suppressing warnings or skipping the contained headers in dependency calculations (see compiler documentation). Additionally, system include directories are searched after normal include directories regardless of the order specified.
Therefore, in https://gitlab.dune-project.org/core/dune-common/-/blob/1437ba8a2840120ef48d563f798134cbb66db70c/cmake/modules/DuneEnableAllPackages.cmake#L301,
there could be an option to include these as SYSTEM header or even as default?
I'm not sure if this will break.
Maybe there is another mechanism less intrusive I'm unaware of.
Nevertheless, this is a show stopper for me to enable warning, since currently the warning pollution of dependencies is too big.
Any thoughts on this @simon.praetorius @santiago.ospina ? Thanks!
https://gitlab.dune-project.org/core/dune-common/-/issues/352
DUNE ENV 2.0
2024-02-22T15:19:29Z
Robert K
DUNE ENV 2.0
The desire is to build DUNE without automatic virtual env creation and pip install.
Related to
- !222.
- !960
- !1104
The desire is to build DUNE without automatic virtual env creation and pip install.
Related to
- !222.
- !960
- !1104
https://gitlab.dune-project.org/core/dune-common/-/issues/350
Vc code not compiling
2023-12-21T16:12:35Z
Christoph Grüninger
Vc code not compiling
The expensive tests were not run for !1275, so I decided to install Vc and test locally. To my surprise I cannot compile dune-common's Vc/SIMD tests, as they fail with ambiguous overloads. This happens for master and both GCC 13 and Clan...
The expensive tests were not run for !1275, so I decided to install Vc and test locally. To my surprise I cannot compile dune-common's Vc/SIMD tests, as they fail with ambiguous overloads. This happens for master and both GCC 13 and Clang 16. (Vc 1.4.3, most recent).
Is anybody using Vc? Is it me or my new compilers? What happyenden to the runners of the expensive tests?
I have to yet tried the 2.9 branch, maybe we have to fix this prior to the upcoming patch release 2.9.1.
DUNE 2.10.0
https://gitlab.dune-project.org/core/dune-common/-/issues/349
debugaligntest fails with Clang 17 branch
2023-12-21T16:12:25Z
Christoph Grüninger
debugaligntest fails with Clang 17 branch
`debugaligntest` fails with Clang 16 (and the not yet released version of Clang 17). With g++ 13 I see no issue. Any idea what went wrong?
```
> ./debugaligntest
CHECK FAILED(default construct): misalignment not detected for Dune::Al...
`debugaligntest` fails with Clang 16 (and the not yet released version of Clang 17). With g++ 13 I see no issue. Any idea what went wrong?
```
> ./debugaligntest
CHECK FAILED(default construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<bool, 32ul>
CHECK FAILED(destruct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<bool, 32ul>
CHECK FAILED(move construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<bool, 32ul>
CHECK FAILED(copy construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<bool, 32ul>
CHECK FAILED(default construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<char, 32ul>
CHECK FAILED(destruct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<char, 32ul>
CHECK FAILED(move construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<char, 32ul>
CHECK FAILED(copy construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<char, 32ul>
CHECK FAILED(default construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<signed char, 32ul>
CHECK FAILED(destruct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<signed char, 32ul>
CHECK FAILED(move construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<signed char, 32ul>
CHECK FAILED(copy construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<signed char, 32ul>
CHECK FAILED(default construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<unsigned char, 32ul>
CHECK FAILED(destruct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<unsigned char, 32ul>
CHECK FAILED(move construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<unsigned char, 32ul>
CHECK FAILED(copy construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<unsigned char, 32ul>
CHECK FAILED(default construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<short, 32ul>
CHECK FAILED(destruct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<short, 32ul>
CHECK FAILED(move construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<short, 32ul>
CHECK FAILED(copy construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<short, 32ul>
CHECK FAILED(default construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<unsigned short, 32ul>
CHECK FAILED(destruct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<unsigned short, 32ul>
CHECK FAILED(move construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<unsigned short, 32ul>
CHECK FAILED(copy construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<unsigned short, 32ul>
CHECK FAILED(default construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<int, 32ul>
CHECK FAILED(destruct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<int, 32ul>
CHECK FAILED(move construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<int, 32ul>
CHECK FAILED(copy construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<int, 32ul>
CHECK FAILED(default construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<unsigned int, 32ul>
CHECK FAILED(destruct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<unsigned int, 32ul>
CHECK FAILED(move construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<unsigned int, 32ul>
CHECK FAILED(copy construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<unsigned int, 32ul>
CHECK FAILED(default construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<long, 32ul>
CHECK FAILED(destruct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<long, 32ul>
CHECK FAILED(move construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<long, 32ul>
CHECK FAILED(copy construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<long, 32ul>
CHECK FAILED(default construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<unsigned long, 32ul>
CHECK FAILED(destruct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<unsigned long, 32ul>
CHECK FAILED(move construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<unsigned long, 32ul>
CHECK FAILED(copy construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<unsigned long, 32ul>
CHECK FAILED(default construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<long long, 32ul>
CHECK FAILED(destruct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<long long, 32ul>
CHECK FAILED(move construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<long long, 32ul>
CHECK FAILED(copy construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<long long, 32ul>
CHECK FAILED(default construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<unsigned long long, 32ul>
CHECK FAILED(destruct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<unsigned long long, 32ul>
CHECK FAILED(move construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<unsigned long long, 32ul>
CHECK FAILED(copy construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<unsigned long long, 32ul>
CHECK FAILED(default construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<wchar_t, 32ul>
CHECK FAILED(destruct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<wchar_t, 32ul>
CHECK FAILED(move construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<wchar_t, 32ul>
CHECK FAILED(copy construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<wchar_t, 32ul>
CHECK FAILED(default construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<char16_t, 32ul>
CHECK FAILED(destruct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<char16_t, 32ul>
CHECK FAILED(move construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<char16_t, 32ul>
CHECK FAILED(copy construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<char16_t, 32ul>
CHECK FAILED(default construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<char32_t, 32ul>
CHECK FAILED(destruct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<char32_t, 32ul>
CHECK FAILED(move construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<char32_t, 32ul>
CHECK FAILED(copy construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<char32_t, 32ul>
CHECK FAILED(default construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<float, 32ul>
CHECK FAILED(destruct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<float, 32ul>
CHECK FAILED(move construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<float, 32ul>
CHECK FAILED(copy construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<float, 32ul>
CHECK FAILED(default construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<double, 32ul>
CHECK FAILED(destruct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<double, 32ul>
CHECK FAILED(move construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<double, 32ul>
CHECK FAILED(copy construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<double, 32ul>
CHECK FAILED(default construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<long double, 32ul>
CHECK FAILED(destruct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<long double, 32ul>
CHECK FAILED(move construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<long double, 32ul>
CHECK FAILED(copy construct): misalignment not detected for Dune::AlignedNumberImpl::AlignedNumber<long double, 32ul>
TEST FAILED: 72/8103 checks failed in this test.
```
DUNE 2.10.0
https://gitlab.dune-project.org/core/dune-common/-/issues/348
Storage duration of MPIHelper singleton
2024-02-22T09:53:43Z
Santiago Ospina De Los Ríos
sospinar@gmail.com
Storage duration of MPIHelper singleton
The following discussion from !1274 should be addressed:
- [ ] @andreas.dedner started a [discussion](https://gitlab.dune-project.org/core/dune-common/-/merge_requests/1274#note_129413): (+4 comments)
> Just wanted to mention that...
The following discussion from !1274 should be addressed:
- [ ] @andreas.dedner started a [discussion](https://gitlab.dune-project.org/core/dune-common/-/merge_requests/1274#note_129413): (+4 comments)
> Just wanted to mention that we ran into problem with this singleton pattern with the python bindings - in fact the issue can come up with any code that combines multiple object files I believe.
> The issue has to do with 'semantic interposition` and leads to segfaults with clang:
>
> Basically, clang did not guarantee that these singleton are unique between different units. So the variable would be initialized in one object file but there might be a separate version of the `singleton` in a different unit that was not initialized. That obviously leads to problems. I looked at this some time ago and by default gcc assumes semantic-interposition if not explicitly turned off (that is the ELF default). The version of clang that was available at the time (2y ago but don't have the version) provided no semantic-interposition by default and unfortunately had no flag to turn it on.
>
> We solved the problem in dune-fem by putting the singleton into `libdunefem` which all other modules link against. That makes the singleton truly unique. The hard ones we had were template dependent singletons...
>
> The following blog post that I found at the time has been updated in 2022 and apparently clang has a flag now to turn on `semantic interposition`: https://maskray.me/blog/2021-05-09-fno-semantic-interposition
> So we could probably require that flag for python bindings to avoid the issue or possibly the singleton can be instantiated in `libdunecommon`.
Andreas Dedner
Andreas Dedner
https://gitlab.dune-project.org/core/dune-common/-/issues/345
Document how to make interdependent targets on different namespaces
2023-10-05T11:35:07Z
Santiago Ospina De Los Ríos
sospinar@gmail.com
Document how to make interdependent targets on different namespaces
The following discussion from !1247 should be addressed:
- [ ] @simon.praetorius started a [discussion](https://gitlab.dune-project.org/core/dune-common/-/merge_requests/1247#note_129022): (+13 comments)
> Unfortunately, this does...
The following discussion from !1247 should be addressed:
- [ ] @simon.praetorius started a [discussion](https://gitlab.dune-project.org/core/dune-common/-/merge_requests/1247#note_129022): (+13 comments)
> Unfortunately, this does not work properly. If one target depends on the other target (e.g. via `target_link_libraries`), the dependency must be included before. Otherwise I get the fatal error "The following imported targets are referenced, but are missing..." automatically generated in the target file. Not sure how to fix this properly. I have to create the libraries in the right order, I think, and then it works. But this is not the way to want to resolve dependencies.
https://gitlab.dune-project.org/core/dune-common/-/issues/343
Follow-up from "[cleanup][OverloadCompilerFlags] Move compiler flag overloadi...
2023-09-18T21:16:45Z
Andreas Dedner
Follow-up from "[cleanup][OverloadCompilerFlags] Move compiler flag overloading to user script."
The following discussion from !1251 should be addressed:
- [ ] @santiago.ospina started a [discussion](https://gitlab.dune-project.org/core/dune-common/-/merge_requests/1251#note_128976): (+26 comments)
> @robert.kloefkorn There i...
The following discussion from !1251 should be addressed:
- [ ] @santiago.ospina started a [discussion](https://gitlab.dune-project.org/core/dune-common/-/merge_requests/1251#note_128976): (+26 comments)
> @robert.kloefkorn There is some errors in the CI for [dune-istl](https://gitlab.dune-project.org/core/dune-istl/-/jobs/491718) and [dune-ugrid](https://gitlab.dune-project.org/staging/dune-uggrid/-/jobs/490354) wrt the configuration of dune-py. The only change in between the failing pipelines seems to be this MR. The error seems to be that the generated cmake has/non-existing wrong parameters.
https://gitlab.dune-project.org/core/dune-common/-/issues/338
[Python] Create Python bindings when Python package has different name to dun...
2023-09-22T14:01:44Z
Alexander Müller
[Python] Create Python bindings when Python package has different name to dune module
I'm trying and struggling currently to create working python bindings for python bindings, where the pypi package should have a different name to the dune module and to the actual python package.
I'm using all 2.9 branches
I.e. my dune...
I'm trying and struggling currently to create working python bindings for python bindings, where the pypi package should have a different name to the dune module and to the actual python package.
I'm using all 2.9 branches
I.e. my dune-module is called "ikarus" which is not available at pypi. Thus, as a workaround I want to name the pypi package `pyikarus`.
Thus I want the behaviour to be `pip install pyikarus` and in the python code `import ikarus` and my cpp folder structure should also be with `ikarus`.
I investigated, and my current setup is:
In the top-level `setup.py` I have
```python
...
metadata["name"] = "pyikarus"
setup(**metadata)
```
In `python/ikarus/setup.py.in` I have
```python
REQUIRED_PACKAGES = '${RequiredPythonModules}'.replace(';',' ').split(' ')
setup(
name="pyikarus",
description="${ProjectDescription}",
version="${ProjectVersionString}",
author="${ProjectAuthor}",
author_email="${ProjectMaintainerEmail}",
packages=find_packages(exclude=["docs/*"]),
zip_safe=0,
package_data={"": ["*.so"], "pyikarus": ["data/*.cmake"]},
install_requires=REQUIRED_PACKAGES,
include_package_data=True,
)
```
and in `python/ikarus/CMakeLists.txt`
```cmake
add_subdirectory(ikarus)
set(CXX_MAX_STANDARD 20)
set(PYPI_PACKAGE_NAME "pyikarus")
dune_python_configure_bindings(
PATH
"."
PACKAGENAME
${PYPI_PACKAGE_NAME}
CMAKE_METADATA_FLAGS
DUNE_OPTS_FILE
CXX_MAX_STANDARD
)
if(POLICY CMP0087)
cmake_policy(SET CMP0087 NEW)
endif()
```
There are several points where I have to state `ikarus` or `pyikarus`. Brute forcing some variants of these options always returns errors.
Thus, if I install my uploaded package from pypi and try to compile my code with `dune-py`, I get two different error messages.
If I pass to `dune_python_configure_bindings` above the name `pyikarus`, I get errors like
<details>
<summary>pyikarus error message</summary>
```python
---------------------------------------------------------------------------
CompileError Traceback (most recent call last)
Cell In[35], line 1
----> 1 svk = iks.StVenantKirchhoff(emodul=1000,nu=0.3)
3 svkPS = svk.asPlainStress()
File /dune/dune-common/build-cmake/dune-env/lib/python3.10/site-packages/ikarus/materials.py:33, in materialConstructorDecorator..wrapper(emodul, nu)
31 includes += ["ikarus/python/finiteElements/materials/material.hh"]
32 moduleName = func.__name__+"_" + hashIt(element_type)
---> 33 module = generator.load(
34 includes=includes,
35 typeName=element_type,
36 moduleName=moduleName
37 )
39 # dispatcher = { 'NeoHooke' : NeoHooke, 'StVenantKirchhoff' : StVenantKirchhoff}
42 return eval("module."+func.__name__+"(emodul,nu)")
File /dune/dune-common/build-cmake/python/dune/generator/generator.py:172, in SimpleGenerator.load(self, includes, typeName, moduleName, extraCMake, defines, preamble, postscript, options, bufferProtocol, dynamicAttr, baseClasses, holder, *args)
168 for nr, (tn, a, o, b, d, bc, h) in enumerate( zip(typeName, args, options, bufferProtocol, dynamicAttr, baseClasses, holder) ):
169 source += self.main(nr, includes, tn, *a, options=o,
170 bufferProtocol=b, dynamicAttr=d,
171 baseClasses=bc, holder=h)
--> 172 return self.post(moduleName, source, postscript, extraCMake)
File /dune/dune-common/build-cmake/python/dune/generator/generator.py:124, in SimpleGenerator.post(self, moduleName, source, postscript, extraCMake)
122 # make sure to reload the builder here in case it got updated
123 from . import builder
--> 124 module = builder.load(moduleName, source, self.typeName[0], extraCMake)
126 return module
File /dune/dune-common/build-cmake/python/dune/generator/cmakebuilder.py:382, in Builder.load(self, moduleName, source, pythonName, extraCMake)
380 module = sys.modules.get("dune.generated." + moduleName)
381 if module is None:
--> 382 self._buildModule( moduleName, source, pythonName, extraCMake )
384 ## TODO remove barrier here
385 comm.barrier()
File /dune/dune-common/build-cmake/python/dune/generator/cmakebuilder.py:732, in MakefileBuilder._buildModule(self, moduleName, source, pythonName, extraCMake)
729 # check return code
730 if exit_code > 0:
731 # retrieve stderr output
--> 732 raise CompileError(buffer_to_str(stderr))
CompileError: /dune/dune-common/build-cmake/dune-env/.cache/dune-py/python/dune/generated/StVenantKirchhoff_de7f96cf2f383220c197a4a670079062.cc:10:10: fatal error: ikarus/finiteElements/mechanics/materials.hh: No such file or directory
10 | #include
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
```
</details>
Thus, the underlying cmake script does maybe not include the correct subdirectories to make the headers findable.
If I pass to `dune_python_configure_bindings` above the name `ikarus` I get errors like
<details>
<summary>pyikarus error message</summary>
```python
---------------------------------------------------------------------------
CompileError Traceback (most recent call last)
Cell In[4], line 1
----> 1 svk = iks.StVenantKirchhoff(emodul=1000,nu=0.3)
3 svkPS = svk.asPlainStress()
File /dune/dune-common/build-cmake/dune-env/lib/python3.10/site-packages/ikarus/materials.py:33, in materialConstructorDecorator..wrapper(emodul, nu)
31 includes += ["ikarus/python/finiteElements/materials/material.hh"]
32 moduleName = func.__name__+"_" + hashIt(element_type)
---> 33 module = generator.load(
34 includes=includes,
35 typeName=element_type,
36 moduleName=moduleName
37 )
39 # dispatcher = { 'NeoHooke' : NeoHooke, 'StVenantKirchhoff' : StVenantKirchhoff}
42 return eval("module."+func.__name__+"(emodul,nu)")
File /dune/dune-common/build-cmake/python/dune/generator/generator.py:172, in SimpleGenerator.load(self, includes, typeName, moduleName, extraCMake, defines, preamble, postscript, options, bufferProtocol, dynamicAttr, baseClasses, holder, *args)
168 for nr, (tn, a, o, b, d, bc, h) in enumerate( zip(typeName, args, options, bufferProtocol, dynamicAttr, baseClasses, holder) ):
169 source += self.main(nr, includes, tn, *a, options=o,
170 bufferProtocol=b, dynamicAttr=d,
171 baseClasses=bc, holder=h)
--> 172 return self.post(moduleName, source, postscript, extraCMake)
File /dune/dune-common/build-cmake/python/dune/generator/generator.py:124, in SimpleGenerator.post(self, moduleName, source, postscript, extraCMake)
122 # make sure to reload the builder here in case it got updated
123 from . import builder
--> 124 module = builder.load(moduleName, source, self.typeName[0], extraCMake)
126 return module
File /dune/dune-common/build-cmake/python/dune/generator/cmakebuilder.py:374, in Builder.load(self, moduleName, source, pythonName, extraCMake)
370 def load(self, moduleName, source, pythonName, extraCMake=None):
371 # check if we need to initialize dune-py either because
372 # this is the first call to load or because an external module with metadata has been registered
373 if not self.initialized or not self.externalPythonModules == getExternalPythonModules():
--> 374 self.initialize()
376 # check whether module is already compiled and build it if necessary
377 # (only try to build module on rank 0!)
378 # TODO replace if rank with something better and remove barrier further down
379 if comm.rank == 0:
File /dune/dune-common/build-cmake/python/dune/generator/cmakebuilder.py:616, in MakefileBuilder.initialize(self)
615 def initialize(self):
--> 616 super().initialize()
617 # check that the compile script is available
618 script = os.path.join(self.generated_dir,"buildScript.sh")
File /dune/dune-common/build-cmake/python/dune/generator/cmakebuilder.py:213, in Builder.initialize(self)
211 self.cacheExternalModules()
212 # create dune-py module
--> 213 self.build_dunepy_from_template(self.dune_py_dir)
214 # create tag file so that dune-py is not rebuilt on the next build
215 open(tagfile, 'a').close()
File /dune/dune-common/build-cmake/python/dune/generator/cmakebuilder.py:472, in MakefileBuilder.build_dunepy_from_template(dunepy_dir, force)
468 if not useNinja:
469 # call base class dunepy_from_template (re-initialize)
470 force = Builder.generate_dunepy_from_template(dunepy_dir, force=True)
--> 472 Builder.callCMake(["cmake"] + defaultCMakeFlags() + ["."],
473 cwd=dunepy_dir,
474 infoTxt="Configuring dune-py with CMake (make)",
475 active=True, # print details anyway
476 )
478 # obtain bash and make command generated by cmake
479 # when ninja is used makeCmd is the ninja command
480 try:
File /dune/dune-common/build-cmake/python/dune/generator/cmakebuilder.py:263, in Builder.callCMake(cmake_args, cwd, infoTxt, verbose, active, logLevel, env)
260 if cmake.returncode > 0:
261 # retrieve stderr output
262 print(stderr,stdout)
--> 263 raise CompileError(buffer_to_str(stderr))
265 return stdout, stderr
CompileError: ---- LOCK
---- ACQUIRED
-- Generating CXX compiler script for CXXFLAGS overloading on command line
---- RELEASE
---- DONE
CMake Error at /dune/dune-common/build-cmake/dune-env/lib/cmake/ikarus/ikarus-targets.cmake:61 (set_target_properties):
The link interface of target "ikarus" contains:
autodiff::autodiff
but the target was not found. Possible reasons include:
* There is a typo in the target name.
* A find_package call is missing for an IMPORTED target.
* An ALIAS target is missing.
Call Stack (most recent call first):
/dune/dune-common/build-cmake/dune-env/lib/cmake/ikarus/ikarus-config.cmake:55 (include)
/dune/dune-common/cmake/modules/DuneModuleDependencies.cmake:170 (find_package)
/dune/dune-common/cmake/modules/DuneModuleDependencies.cmake:289 (find_dune_package)
/dune/dune-common/cmake/modules/DuneModuleDependencies.cmake:336 (dune_process_dependency_leafs)
/dune/dune-common/cmake/modules/DuneModuleDependencies.cmake:46 (dune_create_dependency_leafs)
/dune/dune-common/cmake/modules/DuneProject.cmake:90 (dune_create_dependency_tree)
CMakeLists.txt:106 (dune_project)
```
</details>
which indicates by the linking error, that in some cmake file the linking is not done correctly, thus indicating the first version with `pyikarus` is better? But anyway the headers are not findable in this first case. Changes in the files in `python/ikarus/` does not change the error messages.
Can someone please point me to the correct changes. I'm willing to create then a proper MR to enable this functionality, if this is of interest. Otherwise, if this should already work can someone correct my mistakes?
The current folder structure can be found at https://github.com/ikarus-project/ikarus/tree/feature/pythonBindingsV3.
Thanks, and any help is very much appreciated.
Alexander Müller
Alexander Müller
https://gitlab.dune-project.org/core/dune-common/-/issues/332
How to add include directories to dune targets
2024-02-23T16:12:30Z
Santiago Ospina De Los Ríos
sospinar@gmail.com
How to add include directories to dune targets
### Issue
In order to move to a target based build system, our targets need to be equipped with the necessary include directories so that we can stop relying on project-wise include commands (i.e. `target_include_directories` vs `includ...
### Issue
In order to move to a target based build system, our targets need to be equipped with the necessary include directories so that we can stop relying on project-wise include commands (i.e. `target_include_directories` vs `include_directories`). Currently, we are using `include_directories` in `dune_project()` so that all the targets in the project see the header files of all dependencies and the project itself. This needs to be removed in favor of the target based approach (i.e. `target_include_directories`). The question here is: what is the best alternative to enforce `target_include_directories` in all dune modules?
### Alternatives
_@simon.praetorius made a great summary in https://gitlab.dune-project.org/core/dune-common/-/merge_requests/1249#note_127842_
There are some ideas how to move the include_directories property from a global property into targets:
1. Manually setting the include dir to a target. Pro: it is explicit and works right now. Con: Every module needs to create a library. It is quiet a complicated cmake statement to get right. It should actually be something like
```cmake
target_include_directories(<target> [INTERFACE]
$<BUILD_INTERFACE:${PROJECT_BINARY_DIR}>
$<BUILD_INTERFACE:${PROJECT_SOURCE_DIR}>
$<INSTALL_INTERFACE:${CMAKE_INSTALL_INCLUDEDIR}>)
```
2. Extending the function `dune_project(<target> [INTERFACE]...)` by additional arguments to directly create a module library and set the include directories on this. Pro: expects only little change in the code, can be backwards compatible. Con: now `dune_project` has two functions, configuring the module and creating a library. See !944
3. Adding a helper function just for the purpose of semiautomatically configuring a module library, e.g., `dune_add_module_library(<target> [INTERFACE]...)`. Pro: clear separation of functionality `dune_project` vs. `dune_add_library` but still some automatism. Con: Not yet clear about backwards compatible implementation.
CMake Modernization
Simon Praetorius
Simon Praetorius
https://gitlab.dune-project.org/core/dune-common/-/issues/330
[discussion] drop or simplify pip & venv magic during configure
2023-04-27T14:33:51Z
Christian Engwer
[discussion] drop or simplify pip & venv magic during configure
As we have seen in !1235, the automagic handling of python dependencies leads to a couple of strange problems, e.g. packages installed automatically by `cmake` are not always found.
I would like to start a discussion on how we can simpl...
As we have seen in !1235, the automagic handling of python dependencies leads to a couple of strange problems, e.g. packages installed automatically by `cmake` are not always found.
I would like to start a discussion on how we can simply this whole cmake-venv-pip integration. My provocative statement is that it leads to more problems than it solves and that a clear error message about missing dependencies would be far more helpful.
Opinions?
https://gitlab.dune-project.org/core/dune-common/-/issues/329
Build requirements are not properly documented
2023-09-23T10:46:42Z
Carsten Gräser
graeser@math.fau.de
Build requirements are not properly documented
Currently the build instructions in `README.md` and `INSTALL` are missing a lot of very important information that should be added:
* An internet connection is needed because `pip` is used to download python packages.
* When no connecti...
Currently the build instructions in `README.md` and `INSTALL` are missing a lot of very important information that should be added:
* An internet connection is needed because `pip` is used to download python packages.
* When no connection is available, one gets several warnings and error messages.
However, the build seems to terminate successfully. Furthermore CMake says that the optional package `Python3`has been found.
Either this is not a problem and the build is successful, then we should not scare users with these error messages. Or this
means that the python-bindings cannot be used than this should be properly reported to the user.
* If the internet connection is available, it is used to download executable code from third party locations that is executed later.
* Since the required python modules are not documented it's also not possible that the user provides them on his own
to avoid this dangerous behavior. Notice that even if the python packages mentioned in the error messages are installed
globally, the attempt to download them and the mentioned error messages persists.
BTW:
One can argue that downloading and executing code from third party locations is a no-go unless the user explicitly opts in.
This ticket is not about changing this behavior now. But we should at least be honest and clearly document this and tell the
users how it can be disabled.
Markus Blatt
Markus Blatt
https://gitlab.dune-project.org/core/dune-common/-/issues/328
[Python] JIT compilation and python tests with installed modules
2023-03-01T11:23:17Z
Alexander Müller
[Python] JIT compilation and python tests with installed modules
Hello everyone,
First, I'd like to thank everyone for Dune in general, and since this is about Python bindings, especially the guys who made it possible to JIT compile Python bindings.
I want to use Python bindings and JIT compilation ...
Hello everyone,
First, I'd like to thank everyone for Dune in general, and since this is about Python bindings, especially the guys who made it possible to JIT compile Python bindings.
I want to use Python bindings and JIT compilation in my own module.
To tests this I created the [repo](https://gitlab.dune-project.org/alexander.mueller/testpythonbindings).
I want to execute this within a Docker container with _installed_ modules.
For a clean docker container I think its better to have it installed, since otherwise it is annoying to help CMake to find every dune-module. Furthermore, sicne i want to do this in a docker container pythons venvs are not really needed, aren't they?
It works with uninstalled modules right now but with installed modules I get different kinds of the same error message, namely
```python
1: Comparing build directories of installed dune modules with given build directories
1: build dir /usr/local/lib/cmake/dune-common for module dune-common is expected to be unique across the given metadata - found /dune/dune-common/build-cmake
1: Dune python package could not be found.
```
Thus, I assume it still wants to refer to the build folders. This error occurs independent, if I delete those directories for building.
See also discussion below.
Thanks in advance.
@robert.kloefkorn : Do you think this refactor of this issues text and title is appropriate and makes the problem more clear for others?
<details>
<summary>Initial issue text</summary>
Hello everyone,
First, I'd like to thank everyone for Dune in general, and since this is about Python bindings, especially the guys who made it possible to JIT compile Python bindings.
## TL;DR
I want to create a docker container, where I can develop my own dune module.
It was easy to to this for doing C++ and running the C++ ctests.
I don't really now, if my problems arise due to a wrong configured docker container or wrongly configured project files.
When I enabled Python bindings several errors occur when I run, e.g.
```shell
dunecontrol --only=projectname all
cd build-cmake
ctest --verbose
```
This yields usually several different version of the error involving `[..] is expected to be unique across the given metadata. Got [..]`, when `dune-py` is created.
Usually it fails with
```python
1: Comparing build directories of installed dune modules with given build directories
1: build dir /usr/local/lib/cmake/dune-common for module dune-common is expected to be unique across the given metadata - found /dune/dune-common/build-cmake
1: Dune python package could not be found.
```
or later
```python
1: builddirs = metaData.zip_across_modules("DEPS", "DEPBUILDDIRS")
1: File "/tmp/testpythonbindings/cmake-build-debug-dockerex/dune-env/lib/python3.9/site-packages/dune/packagemetadata.py", line 489, in zip_across_modules
1: raise ValueError(f"build dir {v} for module {k} is expected to be unique across the given metadata - found {result[k]}")
1: ValueError: build dir /usr/local/lib/cmake/dune-common for module dune-common is expected to be unique across the given metadata - found /dune/dune-common/build-cmake
```
where it used installed module that refer to some existing or non-existing build directories.
In https://gitlab.dune-project.org/alexander.mueller/testpythonbindings/-/blob/main/projectname/python/test/test1.py
in the line `v = FieldVector([0, 1, 2])`.
How can i create this docker container and setup the repository correclty that this works? What is the correct way to execute the Python tests then with ctest along the C++ tests? I want the dependencies to be installed.
## Long version with several tries
- I struggled for weeks to get the Python bindings to work, and I believe I misunderstood a number of things.
- I read multiple issues as well as MRs, e.g. !960,!1086, #313, #247, and #279.
- I reviewed the [docker containers](https://gitlab.dune-project.org/docker/ci) and the [continuous integration](https://gitlab.dune-project.org/infrastructure/dune-nightly-test)
- I studied how it is accomplished in [dumux](https://git.iws.uni-stuttgart.de/dumux-repositories/dumux) and other places because I wanted a module with its own namespace. The Python bindings for multiple modules are implemented differently. The folder structures may differ slightly. There remain unused files, such as setup.py.in.
Under certain conditions, it was possible to execute my Python tests, but when I added a dependency to dune.module, it stopped working for unknown reasons.
I chose to create this issue, and I'm willing to create a repository of best practices for obtaining up-to-date, functional Python bindings. Specifically, which files are optional if a "dune. [submodule]" submodule is desired?
In addition, it should detail how to execute Python tests from the build directory and use its an installed module.
In addition, this includes the case where the module depends on other Dune modules that are installed (globally) or not, as well as the relationship between Python venv and `run-in-dune-env`.
To pinpoint the errors precisely, I created this MWE repository at https://gitlab.dune-project.org/alexander.mueller/testpythonbindings.
In building this, I was unable to replicate the errors that exist in my actual project.
I cannot even reach the point where my initial errors occurred due to new errors.
In any case, I believe the errors arise from my general lack of understanding.
Possibly my problems are relatively minor, but I cannot see the solution. In this case, I'm hoping someone can assist me. Then this issue can be resolved and the repository for best practices can be removed. Thank you beforehand.
The repository currently contains the following README, which I am including here for the sake of discussion.
The README file contains the following:
# Test runs
First i tried to get the MWE run with the docker images from CI.
## Run python tests locally in the build folder
### Not installed depencencies
Inside the [docker container of 2.9](https://gitlab.dune-project.org/docker/ci/-/blob/master/dune-2.9/Dockerfile) the dune modules are not installed,
since `DUNECI_INSTALL_STAGE` is not set to 1, see [Link](https://gitlab.dune-project.org/docker/ci/-/blob/master/base-common/duneci-install-module#L118-L119), aren't they?
Running the python tests of projectname fails using the following commands. (To get this working i had to set
`set (CMAKE_PREFIX_PATH "/duneci/modules/dune-common/build-cmake")` in the toplevel `CMakeLists.txt`)
```shell
docker container run -it --entrypoint /bin/bash registry.dune-project.org/docker/ci/dune:2.9
git clone https://gitlab.dune-project.org/alexander.mueller/testpythonbindings.git
cd testpythonbindings/
dunecontrol --only=projectname all
cd build-cmake
ctest --verbose
```
`test1.py` and `test2.py` fails with:
```python
ValueError: Key INSTALL_PREFIX is expected to be unique across the given metadata. Got {'/usr/local', '/duneci/install'}
```
<details>
<summary>Full test results</summary>
```shell
duneci@e7fac7d916b9:~/testpythonbindings/build-cmake$ ctest --verbose
UpdateCTestConfiguration from :/duneci/testpythonbindings/build-cmake/DartConfiguration.tcl
Parse Config file:/duneci/testpythonbindings/build-cmake/DartConfiguration.tcl
UpdateCTestConfiguration from :/duneci/testpythonbindings/build-cmake/DartConfiguration.tcl
Parse Config file:/duneci/testpythonbindings/build-cmake/DartConfiguration.tcl
Test project /duneci/testpythonbindings/build-cmake
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 1
Start 1: pytest1
1: Test command: /duneci/testpythonbindings/build-cmake/run-in-dune-env "python" "test1.py"
1: Test timeout computed to be: 3600
1: Comparing build directories of installed dune modules with given build directories
1: DUNE-INFO: Generating dune-py module in /duneci/modules/dune-common/build-cmake/dune-env/.cache/dune-py
1: Help on package projectname:
1:
1: NAME
1: projectname
1:
1: DESCRIPTION
1: # SPDX-FileCopyrightText: Copyright © DUNE Project contributors, see file LICENSE.md in module root
1: # SPDX-License-Identifier: LicenseRef-GPL-2.0-only-with-DUNE-exception
1:
1: PACKAGE CONTENTS
1: _projectname
1:
1: FUNCTIONS
1: add(...) method of builtins.PyCapsule instance
1: add(i: int, j: int) -> int
1:
1: A function which adds two numbers
1:
1: FILE
1: /duneci/testpythonbindings/build-cmake/python/projectname/__init__.py
1:
1:
1: Traceback (most recent call last):
1: File "/duneci/modules/dune-common/build-cmake/python/dune/common/__init__.py", line 86, in FieldVector
1: return globals()[fv](values)
1: KeyError: 'FieldVector_3'
1:
1: During handling of the above exception, another exception occurred:
1:
1: Traceback (most recent call last):
1: File "/duneci/testpythonbindings/projectname/python/test/test1.py", line 17, in <module>
1: v = FieldVector([0, 1, 2])
1: File "/duneci/modules/dune-common/build-cmake/python/dune/common/__init__.py", line 90, in FieldVector
1: cls = _loadVec(includes, typeName).FieldVector
1: File "/duneci/modules/dune-common/build-cmake/python/dune/common/__init__.py", line 74, in _loadVec
1: return generator.load(
1: File "/duneci/modules/dune-common/build-cmake/python/dune/generator/generator.py", line 172, in load
1: return self.post(moduleName, source, postscript, extraCMake)
1: File "/duneci/modules/dune-common/build-cmake/python/dune/generator/generator.py", line 124, in post
1: module = builder.load(moduleName, source, self.typeName[0], extraCMake)
1: File "/duneci/modules/dune-common/build-cmake/python/dune/generator/cmakebuilder.py", line 374, in load
1: self.initialize()
1: File "/duneci/modules/dune-common/build-cmake/python/dune/generator/cmakebuilder.py", line 616, in initialize
1: super().initialize()
1: File "/duneci/modules/dune-common/build-cmake/python/dune/generator/cmakebuilder.py", line 213, in initialize
1: self.build_dunepy_from_template(self.dune_py_dir)
1: File "/duneci/modules/dune-common/build-cmake/python/dune/generator/cmakebuilder.py", line 448, in build_dunepy_from_template
1: force = Builder.generate_dunepy_from_template(dunepy_dir, force=force)
1: File "/duneci/modules/dune-common/build-cmake/python/dune/generator/cmakebuilder.py", line 121, in generate_dunepy_from_template
1: context["install_prefix"] = metaData.unique_value_across_modules("INSTALL_PREFIX")
1: File "/duneci/modules/dune-common/build-cmake/python/dune/packagemetadata.py", line 496, in unique_value_across_modules
1: raise ValueError(f"Key {key} is expected to be unique across the given metadata. Got {values}")
1: ValueError: Key INSTALL_PREFIX is expected to be unique across the given metadata. Got {'/usr/local', '/duneci/install'}
1/2 Test #1: pytest1 ..........................***Failed 0.59 sec
test 2
Start 2: pytest2
2: Test command: /duneci/testpythonbindings/build-cmake/run-in-dune-env "python" "test2.py"
2: Test timeout computed to be: 3600
2: Comparing build directories of installed dune modules with given build directories
2: DUNE-INFO: Generating dune-py module in /duneci/modules/dune-common/build-cmake/dune-env/.cache/dune-py
2: Traceback (most recent call last):
2: File "/duneci/testpythonbindings/projectname/python/test/test2.py", line 20, in <module>
2: grid = dune.grid.structuredGrid(lowerLeft,upperRight,elements)
2: File "/duneci/modules/dune-grid/build-cmake/python/dune/grid/core.py", line 13, in structuredGrid
2: return yaspGrid(domain, dimgrid=len(lower), coordinates="equidistantoffset")
2: File "/duneci/modules/dune-grid/build-cmake/python/dune/grid/_grids.py", line 253, in yaspGrid
2: constructor = equidistantOffsetCoordinates(
2: File "/duneci/modules/dune-grid/build-cmake/python/dune/grid/_grids.py", line 165, in equidistantOffsetCoordinates
2: mod = moduleYaspCoordinates(dim,ctype)
2: File "/duneci/modules/dune-grid/build-cmake/python/dune/grid/_grids.py", line 159, in moduleYaspCoordinates
2: module = builder.load(moduleName, source, "yasp coordinates dim={dim} ctype={ct}".format(ct = ctype, dim = dim))
2: File "/duneci/modules/dune-common/build-cmake/python/dune/generator/cmakebuilder.py", line 374, in load
2: self.initialize()
2: File "/duneci/modules/dune-common/build-cmake/python/dune/generator/cmakebuilder.py", line 616, in initialize
2: super().initialize()
2: File "/duneci/modules/dune-common/build-cmake/python/dune/generator/cmakebuilder.py", line 213, in initialize
2: self.build_dunepy_from_template(self.dune_py_dir)
2: File "/duneci/modules/dune-common/build-cmake/python/dune/generator/cmakebuilder.py", line 448, in build_dunepy_from_template
2: force = Builder.generate_dunepy_from_template(dunepy_dir, force=force)
2: File "/duneci/modules/dune-common/build-cmake/python/dune/generator/cmakebuilder.py", line 121, in generate_dunepy_from_template
2: context["install_prefix"] = metaData.unique_value_across_modules("INSTALL_PREFIX")
2: File "/duneci/modules/dune-common/build-cmake/python/dune/packagemetadata.py", line 496, in unique_value_across_modules
2: raise ValueError(f"Key {key} is expected to be unique across the given metadata. Got {values}")
2: ValueError: Key INSTALL_PREFIX is expected to be unique across the given metadata. Got {'/usr/local', '/duneci/install'}
2/2 Test #2: pytest2 ..........................***Failed 0.60 sec
0% tests passed, 2 tests failed out of 2
Label Time Summary:
python = 1.19 sec*proc (2 tests)
quick = 1.19 sec*proc (2 tests)
Total Test time (real) = 1.19 sec
The following tests FAILED:
1 - pytest1 (Failed)
2 - pytest2 (Failed)
Errors while running CTest
```
</details>
### Installed
We use the same container but install the modules as similar? to https://gitlab.dune-project.org/docker/ci/-/blob/master/base-common/duneci-install-module#L118-L119.
(To get this working i had to set
`set (CMAKE_PREFIX_PATH "/duneci/install/lib/cmake/dune-common")` in the toplevel `CMakeLists.txt`)
and removing the modules folder afterwards yields with the following commands
```shell
docker container run -it --entrypoint /bin/bash registry.dune-project.org/docker/ci/dune:2.9
cd modules/
dunecontrol --only=dune-common make install
cd ..
rm -rf modules/
git clone https://gitlab.dune-project.org/alexander.mueller/testpythonbindings.git
cd testpythonbindings/
/duneci/install/bin/dunecontrol --only=projectname all
```
the following error
```
ERROR: The path "/duneci/modules" given in DUNE_CONTROL_PATH does not exist.
Execution of dunecontrol terminated due to errors!
```
I suppose this is on purpose but why is the installation performed into the folder `/duneci/install/bin`and not into `/usr/bin` or `usr/local/bin`?
Somehow the installed `dune-common` should not depend on the downloaded build folder?
But my Linux knowledge is not good enough here, maybe.
### Test with homegrown MWE docker container
This tests with my own docker container, which installs only `dune-common into `/usr/local/...`
For the container see [`DockerImages/DockerFile`](https://gitlab.dune-project.org/alexander.mueller/testpythonbindings/-/blob/main/DockerImages/DockerFile)
**`test1.py` and `test2.py` fails with:
```shell
docker container run -it --entrypoint /bin/bash rath3t/dunepythonbindings:installedmodules
git clone https://gitlab.dune-project.org/alexander.mueller/testpythonbindings.git
cd testpythonbindings/
dunecontrol --only=projectname all
cd build-cmake
ctest --verbose
```
```python
build dir /usr/local/lib/cmake/dune-common for module dune-common is expected to be unique across the given metadata - found /dune/dune-common/build-cmake
2: Dune python package could not be found.
```
<details>
<summary>Full test results</summary>
```shell
root@e4f98ef9aa1e:/testpythonbindings/build-cmake# ctest --verbose
UpdateCTestConfiguration from :/testpythonbindings/build-cmake/DartConfiguration.tcl
Parse Config file:/testpythonbindings/build-cmake/DartConfiguration.tcl
UpdateCTestConfiguration from :/testpythonbindings/build-cmake/DartConfiguration.tcl
Parse Config file:/testpythonbindings/build-cmake/DartConfiguration.tcl
Test project /testpythonbindings/build-cmake
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 1
Start 1: pytest1
1: Test command: /testpythonbindings/build-cmake/run-in-dune-env "python" "test1.py"
1: Test timeout computed to be: 3600
1: Comparing build directories of installed dune modules with given build directories
1: build dir /usr/local/lib/cmake/dune-common for module dune-common is expected to be unique across the given metadata - found /dune/dune-common/build-cmake
1: Dune python package could not be found.
1/2 Test #1: pytest1 ..........................***Skipped 0.10 sec
test 2
Start 2: pytest2
2: Test command: /testpythonbindings/build-cmake/run-in-dune-env "python" "test2.py"
2: Test timeout computed to be: 3600
2: Comparing build directories of installed dune modules with given build directories
2: build dir /usr/local/lib/cmake/dune-common for module dune-common is expected to be unique across the given metadata - found /dune/dune-common/build-cmake
2: Dune python package could not be found.
2/2 Test #2: pytest2 ..........................***Skipped 0.10 sec
100% tests passed, 0 tests failed out of 2
Label Time Summary:
python = 0.20 sec*proc (2 tests)
quick = 0.20 sec*proc (2 tests)
Total Test time (real) = 0.20 sec
The following tests did not run:
1 - pytest1 (Skipped)
2 - pytest2 (Skipped)
```
</details>
Not removing the dune folder where dune-common is downloaded (commenting `RUN rm -rf dune/`) in `DockerImages/DockerFile` yields the same error.
It can be used via the docker image `rath3t/dunepythonbindings: installedmodulesNotDeletedBuildDir`.
# Questions
## Not installed python tests
- Is this [repository](https://gitlab.dune-project.org/alexander.mueller/testpythonbindings) almost setup correctly in terms of python bindings? I got inspired by `dumux`, since I also want to have a module with its own namespace.
- Why does the above tests complain with a reference to `/dune/dune-common/build-cmake` even when the module is installed and this folder is removed?
- How to run the python tests correctly without installing the module projectname?
- What needs to be present and how (installed dependencies) to circumvent the errors from above?
- Is there a magic combination of running these targets?
```
build_tests
projectname
_projectname
test_python
build_python_tests
```.
I consider `metadata_install_python_package_dune,install_python_package_dune,install_python` as not needed for this question, just from the naming they are used for installing the package, aren't they?
## Installing the local python module to run python tests with installed module
How should the module be installed:
From [dune-project.org/dev/adding_python](https://dune-project.org/dev/adding_python/ )
`pip install -v --pre --log logfile --find-links file://$PWD/dist packagename==0.1`
or using `metadata_install_python_package_dune,install_python_package_dune,install_python` in some combination?
Can I then copy the test above `test1.py` and `test2.py` somewhere and then test it with `python test1.py` or does this still have to happen inside the venv and I have to write
`run-in-dune-env python test1.py`? But where should the file `run-in-dune-env` be in this case, since all build directories are potentially deleted?
</details>