Core Modules issueshttps://gitlab.dune-project.org/groups/core/-/issues2023-06-12T07:28:48Zhttps://gitlab.dune-project.org/core/dune-common/-/issues/266Merge-Conflicts in CHANGELOG.md2023-06-12T07:28:48ZSimon PraetoriusMerge-Conflicts in CHANGELOG.md### Merge Conflicts
Whenever someone adds a new feature, fixes a bug, deprecates a function or removes something from the code, a changelog entry should be created. This actually means, nearly each MR adds a line into the `CHANGELOG.md`....### Merge Conflicts
Whenever someone adds a new feature, fixes a bug, deprecates a function or removes something from the code, a changelog entry should be created. This actually means, nearly each MR adds a line into the `CHANGELOG.md`. If there are two MRs at the same time (what is always the case) and one gets merged, the second cannot be merged without rebasing to the latest commit. This is frustrating and increases the development-time.
Can we solve this problem somehow?
There was a similar discussion going on in the group of gitlab developers, see a summary of this discussion process here: https://about.gitlab.com/blog/2018/07/03/solving-gitlabs-changelog-conflict-crisis/
The solution they propose is an automatic changelog tool. But the steps before are: instead of directly filling up a `CHANGELOG.md`, add entries into separate .yml files in a directory, e.g., `CHANGELOG/unreleased/`. These entries are then merged into one markdown file in the release process. There are some subtleties in this process, but those could be solved.
Another idea would be: Add the changelog entry only after the MR is reviewed, directly before there merge. Then there might be less merge conflicts but we need to remember to do this. It has another consequence: The additional commit of an entry in the CHANGELOG file requires another CI pipeline run. This is a wast of computing resources, but also increases the time for a merge. A workaround of this problem could be to add an exception to the `.gitlab-ci.yml` file that when some listed files are changed only, no pipeline will be created.
### Changelog format
Another aspect I have learned from reading about changelog files is to use a proper changelog format for better following the changes, see, e.g., https://keepachangelog.com/en/1.1.0/ or https://docs.gitlab.com/ee/development/changelog.html. The summary is: sort changelog entries in some categories, e.g.,
- added: New feature
- fixed: Bug fix
- changed: Feature change
- deprecated: New deprecation
- removed: Feature removal
- performance: Performance improvement
- other: Other
What do you think about this? Should we improve the handling of changelogs or is this "work without benefits"?https://gitlab.dune-project.org/core/dune-common/-/issues/264Excessive Warnings/Messages for optional module dependencies2023-03-13T13:25:57ZTimo KochExcessive Warnings/Messages for optional module dependenciesThe current ~~CMake system~~ `dunemodules.lib` script prints a "scary" warning if an __optional__ dependency is not found. As an example, installing `dumux` with only the Dune core modules (which is a perfectly functioning setup) gives r...The current ~~CMake system~~ `dunemodules.lib` script prints a "scary" warning if an __optional__ dependency is not found. As an example, installing `dumux` with only the Dune core modules (which is a perfectly functioning setup) gives right at the beginning of `dunecontrol`:
<details>
<summary>See warnings</summary>
```shell
WARNING: could not find module 'dune-uggrid',
module is also unknown to pkg-config.
Maybe you need to adjust PKG_CONFIG_PATH!
'dune-uggrid' is suggested by dune-grid
Skipping 'dune-uggrid'!
WARNING: could not find module 'dune-alugrid',
module is also unknown to pkg-config.
Maybe you need to adjust PKG_CONFIG_PATH!
'dune-alugrid' is suggested by dumux
Skipping 'dune-alugrid'!
WARNING: could not find module 'dune-foamgrid',
module is also unknown to pkg-config.
Maybe you need to adjust PKG_CONFIG_PATH!
'dune-foamgrid' is suggested by dumux
Skipping 'dune-foamgrid'!
WARNING: could not find module 'dune-uggrid',
module is also unknown to pkg-config.
Maybe you need to adjust PKG_CONFIG_PATH!
'dune-uggrid' is suggested by dumux
Skipping 'dune-uggrid'!
WARNING: could not find module 'dune-functions',
module is also unknown to pkg-config.
Maybe you need to adjust PKG_CONFIG_PATH!
'dune-functions' is suggested by dumux
Skipping 'dune-functions'!
WARNING: could not find module 'opm-common',
module is also unknown to pkg-config.
Maybe you need to adjust PKG_CONFIG_PATH!
'opm-common' is suggested by dumux
Skipping 'opm-common'!
WARNING: could not find module 'opm-grid',
module is also unknown to pkg-config.
Maybe you need to adjust PKG_CONFIG_PATH!
'opm-grid' is suggested by dumux
Skipping 'opm-grid'!
WARNING: could not find module 'dune-subgrid',
module is also unknown to pkg-config.
Maybe you need to adjust PKG_CONFIG_PATH!
'dune-subgrid' is suggested by dumux
Skipping 'dune-subgrid'!
WARNING: could not find module 'dune-spgrid',
module is also unknown to pkg-config.
Maybe you need to adjust PKG_CONFIG_PATH!
'dune-spgrid' is suggested by dumux
Skipping 'dune-spgrid'!
WARNING: could not find module 'dune-mmesh',
module is also unknown to pkg-config.
Maybe you need to adjust PKG_CONFIG_PATH!
'dune-mmesh' is suggested by dumux
Skipping 'dune-mmesh'!
```
</details>
^ Edit: The above messages have been fixed in !1018. The following issues remain:
**The following happened many times already: A new user installs a user module (for example `dumux`). They see these warnings. They think they did something wrong. They start installing _all_ optional dependencies. These optional dependencies often have other optional dependencies, so they end up installing 20 modules. Some of these are even outdated and they end up with a broken installation.**
I personally agree with these users that this all-caps warning is definitely over the top and optional is not even mentioned there. I think from a user perspective something like
```
-- Optional dependency 'dune-uggrid' not found. Installing 'dune-uggrid' is optional.
Installing 'dune-uggrid' will enable the use of unstructured (parallel) grids with multiple geometry types.
For more information, visit https://dune-project.org/modules/dune-uggrid/.
```
The detailed message would of course require that such a help message and a documentation website is registered for a given module. As a first step
```
-- Optional dependency 'dune-uggrid' not found. Installing 'dune-uggrid' is optional.
```
would already be much nicer and may not create the same panic state of having done something wrong as the current message.
**The same goes for CMake feature tests.** If no module-specific feature tests have been found there is loads of scary messages that report on not having found some cmake script in that module and all the places where it searched. Not having module-specific feature tests is completely harmless and very common for user modules, so I don't think there should be any message about this.
**Dune dependencies** Below is an example of *excessive* output in a user module configure step. This recursively lists all the required and suggested Dune module dependencies. This output seems to be useful in case there is some version mismatch. In particular, also notice the scary output for opm-common although this package is completely optional (and not even used/supported by the user module, just a suggestion from an upstream package).
<details>
```shell
Dependencies for dumux-glymphatics: dumux (>= 3.5);dune-common (>= 2.8);dune-foamgrid (>= 2.8);dune-alugrid (>= 2.8);dune-subgrid (>= 2.8)
-- Dependencies for dumux: dune-common (>=2.8);dune-grid (>=2.8);dune-localfunctions (>=2.8);dune-istl (>=2.8)
-- Suggestions for dumux: dune-alugrid (>=2.8);dune-foamgrid (>=2.8);dune-uggrid (>=2.8);dune-functions (>=2.8);opm-common;opm-grid;dune-subgrid (>=2.8);dune-spgrid (>=2.8);dune-mmesh (>=1.2)
-- Dependencies for dune-foamgrid: dune-common (>= 2.8);dune-geometry (>= 2.8);dune-grid (>= 2.8)
-- Dependencies for dune-alugrid: dune-grid (>= 2.9)
-- Dependencies for dune-subgrid: dune-common;dune-geometry;dune-grid
-- Suggestions for dune-subgrid: dune-alugrid;dune-uggrid
-- Dependencies for dune-geometry: dune-common (>= 2.9)
-- Dependencies for dune-grid: dune-geometry (>= 2.9)
-- Suggestions for dune-grid: dune-uggrid (>=2.9)
-- Dependencies for dune-grid: dune-geometry (>= 2.9)
-- Suggestions for dune-grid: dune-uggrid (>=2.9)
-- Dependencies for dune-geometry: dune-common (>= 2.9)
-- Dependencies for dune-grid: dune-geometry (>= 2.9)
-- Suggestions for dune-grid: dune-uggrid (>=2.9)
-- Dependencies for dune-grid: dune-geometry (>= 2.9)
-- Suggestions for dune-grid: dune-uggrid (>=2.9)
-- Dependencies for dune-localfunctions: dune-geometry (>= 2.9)
-- Dependencies for dune-istl: dune-common (>= 2.9)
-- Dependencies for dune-alugrid: dune-grid (>= 2.9)
-- Dependencies for dune-uggrid: dune-common (>= 2.9)
-- Dependencies for dune-alugrid: dune-grid (>= 2.9)
-- Dependencies for dune-foamgrid: dune-common (>= 2.8);dune-geometry (>= 2.8);dune-grid (>= 2.8)
-- Dependencies for dune-uggrid: dune-common (>= 2.9)
-- Dependencies for dune-functions: dune-localfunctions (>= 2.9);dune-grid (>= 2.9);dune-istl (>= 2.9);dune-typetree (>= 2.9)
-- Could NOT find opm-common (missing: opm-common_DIR)
-- No full CMake package configuration support available. Falling back to pkg-config.
-- Found PkgConfig: /usr/local/bin/pkg-config (found version "0.29.2")
-- Checking for module 'opm-common '
-- No package 'opm-common' found
-- Could NOT find opm-grid (missing: opm-grid_DIR)
-- No full CMake package configuration support available. Falling back to pkg-config.
-- Checking for module 'opm-grid '
-- No package 'opm-grid' found
-- Dependencies for dune-subgrid: dune-common;dune-geometry;dune-grid
-- Suggestions for dune-subgrid: dune-alugrid;dune-uggrid
-- Dependencies for dune-spgrid: dune-grid (>= 2.9)
-- Could NOT find dune-mmesh (missing: dune-mmesh_DIR)
-- No full CMake package configuration support available. Falling back to pkg-config.
-- Checking for module 'dune-mmesh>=1.2'
-- No package 'dune-mmesh' found
-- Dependencies for dune-geometry: dune-common (>= 2.9)
-- Dependencies for dune-grid: dune-geometry (>= 2.9)
-- Suggestions for dune-grid: dune-uggrid (>=2.9)
-- Dependencies for dune-alugrid: dune-grid (>= 2.9)
-- Dependencies for dune-uggrid: dune-common (>= 2.9)
-- Dependencies for dune-geometry: dune-common (>= 2.9)
-- Dependencies for dune-grid: dune-geometry (>= 2.9)
-- Suggestions for dune-grid: dune-uggrid (>=2.9)
-- Dependencies for dune-localfunctions: dune-geometry (>= 2.9)
-- Dependencies for dune-istl: dune-common (>= 2.9)
-- Dependencies for dune-typetree: dune-common (>= 2.9)
-- Dependencies for dune-geometry: dune-common (>= 2.9)
-- Dependencies for dune-grid: dune-geometry (>= 2.9)
-- Suggestions for dune-grid: dune-uggrid (>=2.9)
-- Dependencies for dune-uggrid: dune-common (>= 2.9)
-- Dependencies for dune-uggrid: dune-common (>= 2.9)
-- Dependencies for dune-geometry: dune-common (>= 2.9)
-- Dependencies for dune-grid: dune-geometry (>= 2.9)
-- Suggestions for dune-grid: dune-uggrid (>=2.9)
-- Dependencies for dune-uggrid: dune-common (>= 2.9)
-- Dependencies for dune-geometry: dune-common (>= 2.9)
```
</details>
**Alberta example** Lots of output from an optional package. Again, this a transitive suggestion from an upstream module (why do I need to be informed in the downstream configure step?).
<details>
```shell
-- Checking for module 'alberta-grid_1d>=3.0'
-- No package 'alberta-grid_1d' found
-- Checking for module 'alberta-grid_2d>=3.0'
-- No package 'alberta-grid_2d' found
-- Checking for module 'alberta-grid_3d>=3.0'
-- No package 'alberta-grid_3d' found
-- Could NOT find Alberta (set PKG_CONFIG_PATH to include the location of the alberta-grid_[n]d.pc files) (missing: ALBERTA_GRID_PREFIX) (Required is at least version "3.0")
```
</details>
**-- Not enabling torture-tests**
I'm thankful. But yes, alugrid is special...
**Packages not found**
```
Could NOT find ParMETIS (missing: PARMETIS_LIBRARY PARMETIS_INCLUDE_DIR)
```
appears four times in the mentioned user module above. That makes me want to install it. Yet it's not used in the user module at all. Isn't it enough to provide this message in the summary of optional packages not found?https://gitlab.dune-project.org/core/dune-common/-/issues/263use make -j all in dunepy at startup2021-12-01T09:42:28ZAndreas Dedneruse make -j all in dunepy at startupProblem
--------
If a header like `fvector.hh` changes all python JIT compiled modules have to be recompiled. At the moment that is done during the import, i.e., sequentially which can be time consuming.
Solution
--------
During the fi...Problem
--------
If a header like `fvector.hh` changes all python JIT compiled modules have to be recompiled. At the moment that is done during the import, i.e., sequentially which can be time consuming.
Solution
--------
During the first import of ``dune.generator` run make in parallel on all existing modules. Possible approach would be to add all targets to `all` in `CMakeList.txt` in dune-py and then run `make -jN all` with some suitable choice of `N`.https://gitlab.dune-project.org/core/dune-common/-/issues/262test/eigenvaluetest.cc fails on ARM64 machines2023-03-14T12:49:13ZPatrick Jaaptest/eigenvaluetest.cc fails on ARM64 machinesThe Debian build for ARM64 fails due to a too strict check for `long double` in the line
```
if((Av - eigenValues[j]*eigenVectors[j]).two_norm() > dim*std::sqrt(std::numeric_limits<field_type>::epsilon()))
```
where we compare
`6.56817e-...The Debian build for ARM64 fails due to a too strict check for `long double` in the line
```
if((Av - eigenValues[j]*eigenVectors[j]).two_norm() > dim*std::sqrt(std::numeric_limits<field_type>::epsilon()))
```
where we compare
`6.56817e-16 > 5.55112e-17`
I see no trivial way how to improve this check. Therefore I decided to open an issue here.
Full Debian build log here (does not contain useful information):
https://buildd.debian.org/status/fetch.php?pkg=dune-common&arch=arm64&ver=2.8.0%7Erc1-1&stamp=1629831577&raw=0https://gitlab.dune-project.org/core/ci-config/-/issues/1Add Coverage report2021-10-05T16:49:50ZChristian EngwerAdd Coverage report@ansgar suggested to collect coverage information for the test. I think a good approach would be to have an additional CI configuration that collects coverage information, or should we add this to every run?
There are examples on how to...@ansgar suggested to collect coverage information for the test. I think a good approach would be to have an additional CI configuration that collects coverage information, or should we add this to every run?
There are examples on how to integrate coverage reports in GitLab, but I'm unsure whether this is only possible in the enterprise edition.https://gitlab.dune-project.org/core/dune-istl/-/issues/100Compiler warnings with gcc 9.32022-09-23T09:38:50ZTimo KochCompiler warnings with gcc 9.3```
dune-istl/dune/istl/paamg/transfer.hh: In instantiation of 'static void Dune::Amg::Transfer<V, V1, Dune::Amg::SequentialInformation>::prolongateVector(const Dune::Amg::AggregatesMap<V>&, Dune::Amg::Transfer<V, V1, Dune::Amg::Sequenti...```
dune-istl/dune/istl/paamg/transfer.hh: In instantiation of 'static void Dune::Amg::Transfer<V, V1, Dune::Amg::SequentialInformation>::prolongateVector(const Dune::Amg::AggregatesMap<V>&, Dune::Amg::Transfer<V, V1, Dune::Amg::SequentialInformation>::Vector&, Dune::Amg::Transfer<V, V1, Dune::Amg::SequentialInformation>::Vector&, T, const Dune::Amg::SequentialInformation&) [with T1 = double; V = long unsigned int; V1 = Dune::BlockVector<Dune::FieldVector<double, 1>, std::allocator<Dune::FieldVector<double, 1> > >; Dune::Amg::Transfer<V, V1, Dune::Amg::SequentialInformation>::Vector = Dune::BlockVector<Dune::FieldVector<double, 1>, std::allocator<Dune::FieldVector<double, 1> > >]':
dune-istl/dune/istl/paamg/amg.hh:1124:27: required from 'void Dune::Amg::AMG<M, X, S, PI, A>::additiveMgc() [with M = Dune::MatrixAdapter<Dune::BCRSMatrix<Dune::FieldMatrix<double, 1, 1>, std::allocator<Dune::FieldMatrix<double, 1, 1> > >, Dune::BlockVector<Dune::FieldVector<double, 1>, std::allocator<Dune::FieldVector<double, 1> > >, Dune::BlockVector<Dune::FieldVector<double, 1>, std::allocator<Dune::FieldVector<double, 1> > > >; X = Dune::BlockVector<Dune::FieldVector<double, 1>, std::allocator<Dune::FieldVector<double, 1> > >; S = Dune::SeqILU<Dune::BCRSMatrix<Dune::FieldMatrix<double, 1, 1>, std::allocator<Dune::FieldMatrix<double, 1, 1> > >, Dune::BlockVector<Dune::FieldVector<double, 1>, std::allocator<Dune::FieldVector<double, 1> > >, Dune::BlockVector<Dune::FieldVector<double, 1>, std::allocator<Dune::FieldVector<double, 1> > >, 1>; PI = Dune::Amg::SequentialInformation; A = std::allocator<Dune::BlockVector<Dune::FieldVector<double, 1>, std::allocator<Dune::FieldVector<double, 1> > > >]'
dune-istl/dune/istl/paamg/amg.hh:892:9: required from 'void Dune::Amg::AMG<M, X, S, PI, A>::apply(Dune::Amg::AMG<M, X, S, PI, A>::Domain&, const Range&) [with M = Dune::MatrixAdapter<Dune::BCRSMatrix<Dune::FieldMatrix<double, 1, 1>, std::allocator<Dune::FieldMatrix<double, 1, 1> > >, Dune::BlockVector<Dune::FieldVector<double, 1>, std::allocator<Dune::FieldVector<double, 1> > >, Dune::BlockVector<Dune::FieldVector<double, 1>, std::allocator<Dune::FieldVector<double, 1> > > >; X = Dune::BlockVector<Dune::FieldVector<double, 1>, std::allocator<Dune::FieldVector<double, 1> > >; S = Dune::SeqILU<Dune::BCRSMatrix<Dune::FieldMatrix<double, 1, 1>, std::allocator<Dune::FieldMatrix<double, 1, 1> > >, Dune::BlockVector<Dune::FieldVector<double, 1>, std::allocator<Dune::FieldVector<double, 1> > >, Dune::BlockVector<Dune::FieldVector<double, 1>, std::allocator<Dune::FieldVector<double, 1> > >, 1>; PI = Dune::Amg::SequentialInformation; A = std::allocator<Dune::BlockVector<Dune::FieldVector<double, 1>, std::allocator<Dune::FieldVector<double, 1> > > >; Dune::Amg::AMG<M, X, S, PI, A>::Domain = Dune::BlockVector<Dune::FieldVector<double, 1>, std::allocator<Dune::FieldVector<double, 1> > >; Dune::Amg::AMG<M, X, S, PI, A>::Range = Dune::BlockVector<Dune::FieldVector<double, 1>, std::allocator<Dune::FieldVector<double, 1> > >]'
dune-istl/dune/istl/paamg/amg.hh:886:10: required from here
dune-istl/dune/istl/paamg/transfer.hh:119:10: warning: implicitly-declared 'constexpr Dune::Imp::base_array_unmanaged<Dune::FieldVector<double, 1>, std::allocator<Dune::FieldVector<double, 1> > >::RealIterator<Dune::FieldVector<double, 1> >& Dune::Imp::base_array_unmanaged<Dune::FieldVector<double, 1>, std::allocator<Dune::FieldVector<double, 1> > >::RealIterator<Dune::FieldVector<double, 1> >::operator=(const Dune::Imp::base_array_unmanaged<Dune::FieldVector<double, 1>, std::allocator<Dune::FieldVector<double, 1> > >::RealIterator<Dune::FieldVector<double, 1> >&)' is deprecated [-Wdeprecated-copy]
119 | end=fine.end();
In file included from dune-istl/dune/istl/bvector.hh:26,
from dune-istl/dune/istl/bcrsmatrix.hh:18,
from dune-istl/dune/istl/matrixmarket.hh:29,
from dune-istl/dune/istl/owneroverlapcopy.hh:33,
from dumux/dumux/linear/parallelhelpers.hh:32,
from dumux/dumux/assembly/fvassembler.hh:34,
from dumux/test/freeflow/navierstokes/donea/main_momentum.cc:34:
dune-istl/dune/istl/basearray.hh:112:7: note: because 'Dune::Imp::base_array_unmanaged<Dune::FieldVector<double, 1>, std::allocator<Dune::FieldVector<double, 1> > >::RealIterator<Dune::FieldVector<double, 1> >' has user-provided 'Dune::Imp::base_array_unmanaged<B, A>::RealIterator<T>::RealIterator(const Dune::Imp::base_array_unmanaged<B, A>::RealIterator<typename std::remove_const<T>::type>&) [with T = Dune::FieldVector<double, 1>; B = Dune::FieldVector<double, 1>; A = std::allocator<Dune::FieldVector<double, 1> >; typename std::remove_const<T>::type = Dune::FieldVector<double, 1>]'
112 | RealIterator(const RealIterator<ValueType>& it)
| ^~~~~~~~~~~~
```
```
dune-istl/dune/istl/paamg/amg.hh:919:26: warning: implicitly-declared 'Dune::Amg::Hierarchy<Dune::OwnerOverlapCopyCommunication<Dune::bigunsignedint<96>, int>, std::allocator<Dune::BlockVector<Dune::FieldVector<double, 1>, std::allocator<Dune::FieldVector<double, 1> > > > >::LevelIterator<Dune::Amg::Hierarchy<Dune::OwnerOverlapCopyCommunication<Dune::bigunsignedint<96>, int>, std::allocator<Dune::BlockVector<Dune::FieldVector<double, 1>, std::allocator<Dune::FieldVector<double, 1> > > > >, Dune::OwnerOverlapCopyCommunication<Dune::bigunsignedint<96>, int> >& Dune::Amg::Hierarchy<Dune::OwnerOverlapCopyCommunication<Dune::bigunsignedint<96>, int>, std::allocator<Dune::BlockVector<Dune::FieldVector<double, 1>, std::allocator<Dune::FieldVector<double, 1> > > > >::LevelIterator<Dune::Amg::Hierarchy<Dune::OwnerOverlapCopyCommunication<Dune::bigunsignedint<96>, int>, std::allocator<Dune::BlockVector<Dune::FieldVector<double, 1>, std::allocator<Dune::FieldVector<double, 1> > > > >, Dune::OwnerOverlapCopyCommunication<Dune::bigunsignedint<96>, int> >::operator=(const Dune::Amg::Hierarchy<Dune::OwnerOverlapCopyCommunication<Dune::bigunsignedint<96>, int>, std::allocator<Dune::BlockVector<Dune::FieldVector<double, 1>, std::allocator<Dune::FieldVector<double, 1> > > > >::LevelIterator<Dune::Amg::Hierarchy<Dune::OwnerOverlapCopyCommunication<Dune::bigunsignedint<96>, int>, std::allocator<Dune::BlockVector<Dune::FieldVector<double, 1>, std::allocator<Dune::FieldVector<double, 1> > > > >, Dune::OwnerOverlapCopyCommunication<Dune::bigunsignedint<96>, int> >&)' is deprecated [-Wdeprecated-copy]
919 | levelContext.pinfo = matrices_->parallelInformation().finest();
In file included from dune-istl/dune/istl/paamg/matrixhierarchy.hh:13,
from dune-istl/dune/istl/paamg/amg.hh:11,
from dumux/dumux/linear/amgbackend.hh:35,
from dumux/dumux/linear/seqsolverbackend.hh:43,
from dumux/test/freeflow/navierstokes/donea/main_momentum.cc:41:
dumux/dune-istl/dune/istl/paamg/hierarchy.hh:134:9: note: because 'Dune::Amg::Hierarchy<Dune::OwnerOverlapCopyCommunication<Dune::bigunsignedint<96>, int>, std::allocator<Dune::BlockVector<Dune::FieldVector<double, 1>, std::allocator<Dune::FieldVector<double, 1> > > > >::LevelIterator<Dune::Amg::Hierarchy<Dune::OwnerOverlapCopyCommunication<Dune::bigunsignedint<96>, int>, std::allocator<Dune::BlockVector<Dune::FieldVector<double, 1>, std::allocator<Dune::FieldVector<double, 1> > > > >, Dune::OwnerOverlapCopyCommunication<Dune::bigunsignedint<96>, int> >' has user-provided 'Dune::Amg::Hierarchy<T, A>::LevelIterator<T1, T2>::LevelIterator(const Dune::Amg::Hierarchy<T, A>::LevelIterator<typename std::remove_const<T>::type, typename std::remove_const<T1>::type>&) [with C = Dune::Amg::Hierarchy<Dune::OwnerOverlapCopyCommunication<Dune::bigunsignedint<96>, int>, std::allocator<Dune::BlockVector<Dune::FieldVector<double, 1>, std::allocator<Dune::FieldVector<double, 1> > > > >; T1 = Dune::OwnerOverlapCopyCommunication<Dune::bigunsignedint<96>, int>; T = Dune::OwnerOverlapCopyCommunication<Dune::bigunsignedint<96>, int>; A = std::allocator<Dune::BlockVector<Dune::FieldVector<double, 1>, std::allocator<Dune::FieldVector<double, 1> > > >; typename std::remove_const<T1>::type = Dune::OwnerOverlapCopyCommunication<Dune::bigunsignedint<96>, int>; typename std::remove_const<T>::type = Dune::Amg::Hierarchy<Dune::OwnerOverlapCopyCommunication<Dune::bigunsignedint<96>, int>, std::allocator<Dune::BlockVector<Dune::FieldVector<double, 1>, std::allocator<Dune::FieldVector<double, 1> > > > >]'
134 | LevelIterator(const LevelIterator<typename std::remove_const<C>::type,
| ^~~~~~~~~~~~~
```DUNE 2.9.0https://gitlab.dune-project.org/core/dune-common/-/issues/259document imported moduls from new CMake versions and make them usable wiht on...2023-03-13T13:29:39ZMarkus Blattdocument imported moduls from new CMake versions and make them usable wiht only the information from dune-module-config.cmakeApplies to at least FindPkgConfig and FindPython3.
As it is now users might be up for quite some surprises and maintenance for people other than the authors is quite hard.Applies to at least FindPkgConfig and FindPython3.
As it is now users might be up for quite some surprises and maintenance for people other than the authors is quite hard.DUNE 2.9.0https://gitlab.dune-project.org/core/dune-grid/-/issues/133Issues with the GmshReader and GmshWriter2022-09-23T09:29:00ZSimon PraetoriusIssues with the GmshReader and GmshWriter## Summary
This issue is a collection of other issues and merge requests related to the GmshReader. In a long run we might want to update the implementation to https://gitlab.mn.tu-dresden.de/iwr/dune-gmsh4. The short perspective would b...## Summary
This issue is a collection of other issues and merge requests related to the GmshReader. In a long run we might want to update the implementation to https://gitlab.mn.tu-dresden.de/iwr/dune-gmsh4. The short perspective would be to fix some issues with the GmshReader and GmshWriter:
### Issues in the tests
- [ ] #117 (Heap-Buffer-Overflow in gmshtest-onedgrid)
### Boundary and Element data
- [x] #107 (New GmshReader Interface does not provide runtime-flexitbility)
- [ ] #80 (No possibility for load-balancing data attached to `boundarySegmentIndex()`)
- [x] !410 (WIP: Feature/gmsh reader dynamic interface)
- [ ] !409 (WIP: [DynamicGmshReader] Runtime-checked Interface to the GmshReader)
- [ ] !405 (WIP: [GmshReader] Interface for allowing run-time data-read decision)
- [ ] !522 ([gmshreader] Replace deprecation warning by note and todo until better solution is found)
### The Gmsh file format
- [ ] #94 (GmshWriter should add "$End" at end of file)
- [ ] #89 (GmshReader fails to ignore $PhysicalNames block)
### Related to the new implementation
- [ ] #85 (Upgrade GMSH reader to GMSH v4)
- [ ] #23 (Common interface for grid-readers)
- [ ] !90 (WIP: Implementation of a grid reader interface)https://gitlab.dune-project.org/core/dune-grid/-/issues/132Issues with the VTKWriter2023-06-04T09:27:44ZSimon PraetoriusIssues with the VTKWriter## Summary
This issue is a collection of other issues and merge requests related to the VTKWriter. It is suggested in the last developer meeting to include the https://gitlab.dune-project.org/extensions/dune-vtk module into dune-grid, bu...## Summary
This issue is a collection of other issues and merge requests related to the VTKWriter. It is suggested in the last developer meeting to include the https://gitlab.dune-project.org/extensions/dune-vtk module into dune-grid, but this is in the long run. The short perspective would be to fix some issues with the VTKWriter:
### Interplay of VTKWriter with Python
- [ ] #97 (vtk test should use python3)
- [ ] #124 (Memory leak in Python bindings related to VTKWriter.)
### The write method and related issues
- [ ] #95 (Possibility to specifiy output path for .pvd files in VTKSequenceWriter?)
- [ ] #92 (Wrong File references in VTKSequenceWriter)
- [ ] #73 (VtkWriter behaves differently in write() and pwrite() for commSize=1)
- [ ] #168 Tensor data is not supported (but there's no documentation or helpful error message)
### Developement of the new dune-vtk interface
- [ ] #103 (Add support for higher order polynomials to VTKWriter)https://gitlab.dune-project.org/core/dune-common/-/issues/256Installed packages causing cmake trouble for source build.2024-03-02T23:44:47ZRobert KInstalled packages causing cmake trouble for source build.On an Ubuntu 20.04 system when the DUNE core packages are installed (e.g. apt install libdune-grid-dev) then the cmake for a master source tool chain fails with the following error:
CMake Warning at /usr/share/dune/cmake/modules/DunePy...On an Ubuntu 20.04 system when the DUNE core packages are installed (e.g. apt install libdune-grid-dev) then the cmake for a master source tool chain fails with the following error:
CMake Warning at /usr/share/dune/cmake/modules/DunePythonTestCommand.cmake:46 (message):
Unparsed arguments in dune_python_add_test: This often indicates typos!
Call Stack (most recent call first):
dune/python/test/CMakeLists.txt:1 (dune_python_add_test)
CMake Error at /usr/share/dune/cmake/modules/DunePythonTestCommand.cmake:54 (message):
dune_python_add_test: no COMMAND to execute specified!
Call Stack (most recent call first):
dune/python/test/CMakeLists.txt:1 (dune_python_add_test)
This means that the cmake files from the master source tree and the system get mixed up. This seem to only occur when Python bindings are enabled (-DDUNE_ENABLE_PYTHONBINDINGS=ON). However, in general it raises the question whether the mix up always occurred and it was simply not noticed.https://gitlab.dune-project.org/core/dune-common/-/issues/254Generated default CMakeLists.txt2023-12-21T23:51:28ZSimon PraetoriusGenerated default CMakeLists.txt1. In the `CMakeLists.txt` file generated by `duneproject` there are some lines that are hard to understand and not documented:
```cmake
if(NOT (dune-common_DIR OR dune-common_ROOT OR
"${CMAKE_PREFIX_PATH}" MATCHES ".*dune-common.*...1. In the `CMakeLists.txt` file generated by `duneproject` there are some lines that are hard to understand and not documented:
```cmake
if(NOT (dune-common_DIR OR dune-common_ROOT OR
"${CMAKE_PREFIX_PATH}" MATCHES ".*dune-common.*"))
string(REPLACE ${CMAKE_PROJECT_NAME} dune-common dune-common_DIR
${PROJECT_BINARY_DIR})
endif()
```
The meaning is as follows:
- If there is no variable `dune-common_DIR` (or `dune-common_ROOT`) is set, and the `CMAKE_PREFIX_PATH` list does not contain
any directory containing the string `dune-common`
- Then guess the build-dir of dune-common by replacing the module name in the module build-dir by the string "dune-common".
- This constructed build-dir is used as directory to find the dune-common package.
This sounds like a wild construction to me and I always wondered why this is actually necessary.
- If we are using `dunecontrol` then the `dune-common_DIR` is explicitly passed to all built modules.
- If we are not using `dunecontrol` we should pass the directories where to find the modules explicitly instead of wild guessing. (in my opinion)
2. The second strange part of the generate `CMakeLists.txt` file is:
```cmake
list(APPEND CMAKE_MODULE_PATH "${PROJECT_SOURCE_DIR}/cmake/modules"
${dune-common_MODULE_PATH})
```
Here we add the cmake directory of the current dune module and the corresponding directory of dune-common to the search path `CMAKE_MODULE_PATH` variable.
- The module path of `dune-common` should be added automatically inside the `find_package` to `CMAKE_MODULE_PATH`, because this is what the package exports.
- The module specific cmake directory is currently necessary, since automatically the cmake file `Dune<Module>.cmake` is included recursively by all dune modules. I'm hoping to remove this automatic inclusion in the future in favor of an extended find-package description.DUNE 2.10.0https://gitlab.dune-project.org/core/dune-common/-/issues/253What is a matrix?2023-10-17T15:32:05ZSimon PraetoriusWhat is a matrix?## What is a matrix?
While we have CRTP base class `DenseMatrix` specifying the interface for matrices, this base is incomplete or wrong documented and even inconsistent. The specification is also not applied to all matrix-like classes n...## What is a matrix?
While we have CRTP base class `DenseMatrix` specifying the interface for matrices, this base is incomplete or wrong documented and even inconsistent. The specification is also not applied to all matrix-like classes neither in dune-common nor in dune-istl. So, I wanted to open a specific discussion on what is a matrix?
### 1. Matrix is a 2d container of values
On Wikipedia, a matrix is defined as "a rectangular array of numbers (or other mathematical objects)". Thus we have the following requirements:
- (a) A matrix stores elements/entries of a specific type. Seeing matrices as collections, this would mean we have a single element-type. For collections this type is typically denoted by `value_type`.
- "The size of a matrix is defined by the number of rows and columns that it contains". Actually, this definition specifies three size informations:
1. (b) There is a type representing the sizes, typically denoted by `size_type`
2. (c) There is number of rows and number of columns. In Dune this is (for historical reasons) denoted by `N()` and `M()` respectively.
3. (d) There us the *total size* of a matrix. Since a matrix shape is denoted by `n x m`, I would suggest to define the size as the product `n*m`. This would allow to see single-row and single-column matrices as row-vector and column-vector, correspondingly. Compare, e.g., `numpy.matrix.size`, `mtl::size(matrix)`, or `Eigen::Matrix::size`.
- (e) The elements of a matrix should be accessible by indices. Since the row and column indices are in the range `[0,rows)` and `[0,cols)`, its index type should also be `size_type`.
- (f) Following the C-array syntax, elements are accessed by `matrix[i][j]`.
### 2. Matrix is an element of a vector space
Matrices over a field build an algebraic set with vector-space properties:
- matrices for an additive abelian group (commutative group)
- matrices are scalable by scalars of the field type
- Multiplication with scalar is distributive with respect to addition.
These mathematical concepts result in the following requirements for the matrix type `M`:
- There is a field type associate to the matrix, i.e. `Dune::FieldType<M>::field_type` exists.
- There are arithmetic operations for matrices `A` and `B`:
* (g) `A + B`, `A - B`
* (h) `A += B`, `A -= B`
* (i) `-A`, `+A` (optional)
- Multiplication with a scalar `s` of field type:
* (j) `A * s`, `s * A`
* (k) `A *= s`
* (l) `A / s`, `A /= s`
- (m) Additionally, we typically require compound operations: `A.axpy(s, B)`
#### 2.1. Normed space and inner-product space
By defining norms over matrices and inner products, the vector spcae could be turned into a normed
vector space or an inner-product space
- Typical norm for matrices include:
* (n) `frobenius_norm()`
* (o) `forbenius_norm2()` (squared frobenius norm)
* (p) `infinity_norm()`
* (q) `infinity_norm_real()` (simplified infinity norm)
- Inner-products:
* (r) frobenius inner-product `A:B = tr(A^H A)` (not implemented by any matrix)
### 3. Matrix is a linear map
A linear transformation or linear map between vector spaces can be represented by a matrix. The operations of a linear map are typically of the form of matrix-vector products:
- (s) `A.mv(x, y)`: `y = A*x`
- (t) `A.umv(x, y)`: `y+= A*x`
- (u) `A.mmv(x, y)`: `y-= A*x`
- (v) `A.usmv(alpha, x, y)`: `y+= alpha * A*x`
If the matrix additionally provide transposed operations:
- (w) `A.mtv(x, y)`: `y = A^T * x`
- (x) `A.umtv(x, y)`: `y+= A^T * x`
- (y) `A.mmtv(x, y)`: `y-= A^T * x`
- (z) `A.usmtv(alpha, x, y)`: `y+= alpha * A^T * x`
If the matrix additionally provide hermitian operations:
- (A) `A.mhv(x, y)`: `y = A^H * x` (currently not implemented by any matrix)
- (B) `A.umhv(x, y)`: `y+= A^H * x`
- (C) `A.mmhv(x, y)`: `y-= A^H * x`
- (D) `A.usmhv(alpha, x, y)`: `y+= alpha * A^H * x`
### 4. Other operations with matrices
- (E) Matrix multiplication: `A in R^nxm` and `B in R^mxk` then `A * B in R^nxk`
* Matrices must be compatible
* square matrices might form a ring under matrix addition and matrix multiplication
this would allow `A *= B`
- (F) The transposed of a matrix: `A in R^nxm` -> `A^T in R^mxn`.
* Either an operation `A.transpose()` or a free function `transpose(A)`?
* The transposed matrix is itself a matrix
- (G) Trace of a matrix `tr(A) = sum_i A_ii`
* Assumption: matrix is square
* Either an operation `A.trace()` or a free function `trace(A)`?
* Recursive definiton: `tr(A) = sum_i tr(A_ii)`
- (H) Determinant of a matrix
* Assumption: matrix is square
* Easy to define for small matrices
* Generalization using, e.g., Laplace expansion
* Special operation: gramian determinant
- (I) Inversion of a matrix or solution of a linear system
* Assumption: matrix is square
* easy to implement for small matrices
* LU decomposition (or other forms of factorization)
### 5. Matrices can be traversed
Visiting all the (stored) elements of a matrix is a useful operation to generalize some algorithms
- Whats is a proper call for choosing the way/order of traversal?
- It exists `begin()` and `end()` functions.
* Are not documented properly but typically mean traversal of rows.
- Additionally reverse traversal is given by `beforeEnd()` and `beforeBegin()`
* Why not using `rbegin()` and `rend()` as for standard containers?
- Missing functions in the interface:
* Traversal over rows first: e.g. `begin_rows()`, `end_rows()`
* Traversal over columns first: e.g. `begin_cols()`, `end_cols()`
### 6. Special matrix types with more properties
- Sparse matrices:
* Check whether elements exist in the store is required: `A.exists(i,j)`
* Number of stored elements (nonzeros): `A.nonzeroes()`
* Iterators only iterate over stored elements/nonzeros.
* Often direct element-access not implemented (neither const nor mutable)
- Banded/Diagonal matrices:
* Special type of sparse matrix
* Additional access to diagonal(s): `A.diagonal()` (current interface does not allow return off-diagonals)
* Direct access to diagonal entries: `A.diagonal(i)`
- Triangular matrices
- Tridiagonal matrices
## Examples in dune-common
### `DenseMatrix`
A `DenseMatrix` is the CRTB base class for some matrix implementation, but it lacks some of the properties from above. Also
its documentation is partially wrong.
- (c) There are multiple functions returning number of rows/columns: `A.N()` and `A.rows()`, `A.M()` and `A.cols()`, where
`rows() -> mat_rows()` and `cols() -> mat_cols()` call the implementation functions.
- (d) is defined differently. For some reason `A.size()` returns the number of rows.
- (g) arithmetic operations not completely available for matrices, `+A` not implemented (not required)
- (j) and (l) not implemented, except for `A /= s`
- (r) There is no inner product for matrices implemented
- (A) Missing lin-map operation `A.mhv(x,y)`
- (E) binary matrix-matrix product not implemented. For square matrices the mult-assign is implemented from left and
right, i.e., `A.leftmultiply(B)` and `A.rightmultiply(B)`. The general matrix-matrix multiplication is commented out.
- (G) Trace is not implemented.
### `FieldMatrix`
Is an implementation of `DenseMatrix` thus inherits its properties. Some additional comments
- (c) Here `rows` and `cols` are enum constants hiding the interface functions `rows()` and `cols()`
- (g) FieldMatrix implements `A + B` and `A - B`
- (j) and (l) are completely implemented
- (E) Multiplication is implemented as `A * B` and `leftmultiply[any]()` and `rightmultiply[any]()`. Additional operations
are implemented in the namespace `Dune::FMatrixHelp`
### `DynamicMatrix`
Is an implementation of `DenseMatrix` thus inherits its properties.
### `DiagonalMatrix`
- (d) is defined differently. For some reason `A.size()` returns the number of rows.
- (f) The element access is only implemented for diagonal entries, i.e. `matrix[i][j]` throws if `i != j`
- (g), (j) Operations `A + B`, `A - B`, and `-A` not implemented.
- (i) compound assignment operators implemented for scalars: `A += s` and `A -= s` meaning elementwise addition and
subtraction of that scalar.
- (m) Operation `axpy` not implemented.
- (r) There is no inner product for matrices implemented
- (A) Missing lin-map operation `A.mhv(x,y)`
- (E) No matrix-matrix products implemented
- (G) Trace is not implemented.
### `TransposedMatrixWrapper`
The wrapper is created by `transpose(matrix)`, but does not produce a matrix in any sense as described above. It
has just one single operation: `A * B^T`. Note, this wrapper only works for transposing a `FieldMatrix`.
## Test of the interface
There is a test in dune-common: `checkmatrixinterface.hh` but it does not test the complete matrix interface, but just the functions
that were already implemented. Thus, this test is probably written after the implementation of the `DenseMatrix` class and
not before.
An alternative test, that is implemented in terms of separating the various aspects of a matrix, is implemented in !953
in terms of c++20 concepts. These tests can run in the ci jobs for toolchains supporting this new c++ standard and can
report interface failures.
## Major problems with the current interface
The existing matrix implementations in dune-common (but also in dune-istl) only partially fulfil the interface definition
from above and are inconsistent. Major problems are:
- Element access is not a defined property of a matrix. It is only guaranteed to succeed in some matrix implementations.
This is mainly for performance reasons. This argument is a bit weak, because there are other ways to ensure performant
data access, e.g. using iterators or specialized access functions.
- Traversal of matrices is not properly specified and cannot easily be extended to column-major traversal. `begin()` and
`end()` have no knowledge about its traversal direction. There should be a better name for row-wise traversal and
column-wise traversal. Proposal: `begin_rows()` and `begin_cols()`. Then, the user has to explicitly use a range-generator
for the iteration, e.g., `rows(matrix).begin()` and `columns(matrix).begin()`
- The iterators itself are not properly specified. Is the dereferenced iterator again a range? What does it traverse?
- Some matrices implement a single index access `matrix[r]` as a row-view onto the matrix, i.e., this operation returns a
vector-like container representing the `r`th row of the matrix. This is not clearly specified. Is this a required property
of a matrix? What are the costs of this operation? Is the row a propery vector or just a proxy providing minimal access
to the row entries?
- Especially in the implementation of geometries, one needs matrix-matrix multiplication. While this is implemented for
`FieldMatrix`, all the other matrix types lack this operation. But, e.g., `YaspGrid` returns for its jacobian a
`DiagonalMatrix`. Other grid implementations maybe something else. We need `A * B`, but also `A^T * B` and `A * B^T`
for various combinations of matrices. This is partially implemented in some "Helper" namespaces in dune-geometry or
for `FieldMatrix` in `FMatrixHelper` namespace. Multiplication seems not properly designed!
- Sparse matrices are a special type of matrices. The interface should mirror this mathematical property. Actually, sparse
matrices are just matrices with a different storage type for its entries. Thus, sparse matrices should only extend the
matrix interface with methods providing fast access to the stored elements. Either by iterators or something else. Currently,
sparse matrices only implement a partial matrix interface and some methods even have different behaviour, e.g. `matrix[i][j]`
is not guaranteed to succeed, not even for const-access, although the matrix value is clearly specified.
- Views over matrices, like transposed-view or hermitian-view are not properly implemented. They do not provide a matrix
interface although this could easily be provided. One could discuss whether views are read-only or provide mutable
access to the matrix elements as well.
- Currently, we can only test that a matrix supports all the different aspects as described above. Not the individual aspect. But, often an algorithm just needs some properties, e.g., in iterative krylov methods, one needs the linear-map properties for a matrix, for lu-decomposition maybe fast element access. In geometry parametrizations we need matrix-matrix multiplication and trace/Gramian determinant computation. It is required to have individual concept/interface checks.
- Single-row and Single-column matrices are not vectors. This often leads to problems, e.g., there is no `two_norm()` implemented and also `dot()` and `axpy` with another vector does not work, even if the shape would allow this. This problem is that vectors are not matrices and also not compatible in its interface to matrices.https://gitlab.dune-project.org/core/dune-istl/-/issues/99[UMFPack] Allow to set UMFPACK_ORDERING via ParameterTree2021-03-04T09:24:03ZKilian Weishaupt[UMFPack] Allow to set UMFPACK_ORDERING via ParameterTreeWe have recently [found a case](https://git.iws.uni-stuttgart.de/dumux-repositories/dumux/-/issues/999) where a different choice of the UMFPACK_ORDERING method (see page 17 [here](https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=we...We have recently [found a case](https://git.iws.uni-stuttgart.de/dumux-repositories/dumux/-/issues/999) where a different choice of the UMFPACK_ORDERING method (see page 17 [here](https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwjHmorN7JPvAhXQ8qQKHbkgDUkQFjAAegQIARAD&url=https%3A%2F%2Ffossies.org%2Flinux%2FSuiteSparse%2FUMFPACK%2FDoc%2FUMFPACK_UserGuide.pdf&usg=AOvVaw2LJe4yrqT0PhXusgXBc3HB)) decreases the linear solve time by a factor of 4.
It would be very convenient to set the ordering via the ParameterTree, using the new interface.https://gitlab.dune-project.org/core/dune-grid/-/issues/128Documentation of MultipleCodimMultipleGeomTypeMapper is incomplete2021-02-24T13:23:11ZCarsten Gräsergraeser@math.fau.deDocumentation of MultipleCodimMultipleGeomTypeMapper is incompleteSince f4f1a59579ab6b3e5a36f249f6be27ff0114231c the `MultipleCodimMultipleGeomTypeMapper` supports multiple DOFs per entity. The documentation of this feature is rather incomplete. E.g. the class documentation of the mapper and the layout...Since f4f1a59579ab6b3e5a36f249f6be27ff0114231c the `MultipleCodimMultipleGeomTypeMapper` supports multiple DOFs per entity. The documentation of this feature is rather incomplete. E.g. the class documentation of the mapper and the layout class do not mention it and the documentation of the `indices` method is hard to understand (and in fact incorrect) if you do not already know what it does.https://gitlab.dune-project.org/core/dune-common/-/issues/249dune python main file2023-02-10T15:06:34ZAndreas Dednerdune python main fileSuggestion:
-----------
#. Add a `__main__.py` to `python/dune` in `dune-common` which provides an option to produce configuration output. So calling `python -m dune configuration` would generate output that can be added to a bug report...Suggestion:
-----------
#. Add a `__main__.py` to `python/dune` in `dune-common` which provides an option to produce configuration output. So calling `python -m dune configuration` would generate output that can be added to a bug report, containing for example `CMakeCache.txt` from `dune-py`,
`pip list | grep dune` and hash information of the dune modules.
#. Add a `upgrade` command to `__main__` that provides an easy way to patch a dune package without having to go through an extra upload to `pypi` (i.e. for a branch or for testing). Something like this
```
pip install scikit-build
pip install --no-cache-dir --no-deps --upgrade --no-build-isolation -i https://gitlab.dune-project.org/api/v4/projects/???/packages/pypi/simple dune-grid
```
could be executed when calling `python -m dune update -i gitlab grid`https://gitlab.dune-project.org/core/dune-common/-/issues/247Overhaul of the Python-CMake integration2023-03-01T11:23:16ZDominic Kempfdominic.kempf@iwr.uni-heidelberg.deOverhaul of the Python-CMake integrationTL;DR: This issue a summarizes the status quo of our Python/CMake integration and proposes some changes. You might want to have a look at the *High Level Goals* part of the proposal below to get an idea.
With the recent discussion of en...TL;DR: This issue a summarizes the status quo of our Python/CMake integration and proposes some changes. You might want to have a look at the *High Level Goals* part of the proposal below to get an idea.
With the recent discussion of enabling the Python bindings by default, our CMake extension for Python code has become more attention. So far, this extension accommodated the needs of several user groups, but did not do this in a unified, coordinated way. With the prospect of default-enabled Python bindings, additional requirements have arisen that require a bit of reengineering of the extension. This issue summarizes all technical aspects and provides a proposal.
## Prerequisistes: Defining *Installation*
This document will talk a lot about *installation*. It is worth to make some fine distinctions here:
* By *Stack Installation* we refer to a method of installing an entire Dune stack including getting all the core sources. An example of this would be Debian packages or Spack repository. We do not cover this type here, because it is not really affected. In the Python context it is important to note that `pip`-installable Dune packages are also in this category and are therefore not subject of this document, although it seems clearly Python-related.
* By *Global Installation* we refer to the installation method that we know from `make install` in C++: A set of sources and build artifacts is copied into a pre-defined directory structure under a standardized system path prefix.
* By *Local Installation* we refer to the same method as *Global installation* only that the user-given prefix that is not a system path.
* By *Package Installation* we refer to the process that is necessary to execute code in a Python environment without manual adjustments to path variables. Although global/local and Package installation of Python code use the same commands (e.g. `python -m pip install`) this still must be distinguished, as it is also part of the build process.
## Prerequisites: Workflow Stages
It is also worth distinguishing the typical stages in a user/developer workflow.
* *CMake stage*: the project is configured.
* *Build stage*: Compiling C++ sources, potentially also building tests
* *Test Stage*: Running tests on the build produced by the build stage
* *Install stage*: Perform global/local installation
## Status Quo:
Currently, the following Python-related things happen at these stages. You can skip over
this part if it is too technical for you, but I feel like there is a big need for a write-up
of this.
* CMake Stage
* Python is searched on the System. It finds the currently active Python,
which might be the System interpreter or a virtualenv that is activated
at the time CMake runs
* We determine whether the found Python is in a virtualenv
* We search for the Python package `pip`
* If the user enabled `DUNE_PYTHON_VIRTUALENV_SETUP`, a virtualenv is created.
This uses either the `virtualenv` or the `venv` package.
If the build directory tree uses an absolute prefix, this env is located in
`build-prefix/dune-python-env`, otherwise it is placed in the build directory
of `dune-common`. If `dune-common` is globally installed, the Dune module
dependency DAG is traversed to find a build directory - this procedure
potentially breaks if there is a diamond dependency pattern with some modules
of the diamond being globally installed. The created virtualenv is only exposed through
the CMake variable `DUNE_PYTHON_VIRTUALENV_EXECUTABLE`, the Python found by CMake
is still accessible. An `activate` script and `run-in-dune-env.sh` wrapper are
placed in each build directory to give users access to the created environment.
* If the virtualenv was set up, Python packages that are provided by this Dune
module are installed into the virtualenv.
* dune-testtools and modules that depend on it continue to execute Python code
during the CMake run.
* Build Stage
* Python Modules that are added with `dune_add_pybind11_module` are built.
* In dune-codegen, some C++ targets depend on a header that is generated running
Python code in the virtualenv. This is managed by the CMake command `add_custom_command`.
* dune-fufem targets link against the Python libraries to start an embedded
interpreter.
* Test Stage
* As part of `ctest`, Python tests that were added through the `dune_python_add_test`
command are executed.
* If you only want to run the Python tests, you can also do `make test_python`.
* When code using the Python bindings is first executed (likely in the test stage),
it sets up a Dune module called `dune-py` that serves as the playground for code
generation and Just-In-Time compilation. This module is currently not placed in
an isolated per-build location leading to potential compatibility issues between builds.
* Install Stage
* A global/local installation of Python packages is performed via `pip`. In contrast to C++,
the prefix choice depends on the CMake Variable `DUNE_PYTHON_INSTALL_LOCATION=user|system|none`,
where
* `none` disables installation of Python packages
* `system` uses the interpreter found by CMake
* `user` uses `pip install --user`, which installs into `~/.local`
If the variable is not set, it defaults to `system` if the found interpreter
runs in a virtualenv and `none` otherwise. The `user` option is incompatible
with virtual environments.
* A wheel (a standarized binary distribution format for Python packages) is built
for each package and installed into a location under the Dune installation prefix (called a wheelhouse).
This is necessary to allow partially installed Dune stacks: If my module depends
on dune-common - where do I get the Python package `dune.common` from? Answer: from
the Dune wheelhouse under the prefix that was used to install dune-common.
* Dependencies of Python packages are automatically installed by pip using the
Python Package Index: This requires network access. Failing to provide it currently
results in a CMake Error (pip network timeout).
* The Python-specific part of the installation can be run separately by using
the dedicated `make install_python`. This is what is advocated and used for
the Python bindings - although you need to do this before you can run the test
stage.
* All installation rules refer to the Python environment that was active when CMake ran - which
might be different from the Python environment at the time of installation.
If you want to read code for this, it is located in `dune-common/cmake/modules/DunePython*.cmake`.
The main entry point is `DunePythonCommonMacros.cmake`.
## Proposal
I am willing to implement this or a similar solution.
### High Level Goals
* With Dune being and staying a C++ project, the above workflow stages and their order should stay the same.
* *At the build stage, all Python code is package-installed into an isolated per-build environment.* (This was a fundamental outcome of the meeting: In the C++ frame of mind, Python Package Installation is part of building, not installing).
* Building a Dune stack without network access should be successful, though Python parts can be disabled in this build.
* Global/local installation of Python should install all Python packages into the Python environment that was active when CMake ran
* ~~Global/local installation of Python should depend on the Python environment at the time of installation~~ (I really wanted this to work, but thinking about it more I realized that the embedded interpreter case struggles heavily with this)
* Performing a global/local installation of the entire Dune stack (e.g. `dunecontrol make install_python`)
should make the build environment disposable.
* Running Python code should be independent of Dune-specific environment variables. (I added this one after some experiments)
### Technical Details
I think, we already have almost all of the necessary components, they just need to be slightly tweaked.
These are only some core ideas, there is a lot of detail to this that can be discussed
once I provided a sample implementation:
* The virtualenv set up is enabled by default
* The following conditions will lead to all Python aspects being disabled
* Python is not found
* Pip or virtualenv/venv are not found
* Pip operations produce a network timeout (this can still be worked around
by setting up PyPI mirrors, but this should never be part of a standard user workflow)
* After virtualenv set up, the CMake check for Python is performed again, so that
CMake only knows the virtual environment.
* Python Package installations into the virtualenv are formulated as custom targets
that are part of `all` iff the bindings are enabled.
* ~~Package installation is executed in CMake Script Mode (`cmake -P`) to allow searching
and finding the current Python environment in the installation stage.~~
* The embedded interpreter use case should use CMake generator expressions to link
to the system Python after local/global installation.
The only problematic use case that I can think of right now is the dune-testtools
one - but that mostly affects myself. I will find a way to deal with it.https://gitlab.dune-project.org/core/dune-common/-/issues/246Use CMake components when installing2023-11-13T12:12:16ZSantiago Ospina De Los Ríossospinar@gmail.comUse CMake components when installing### Description
While modernising my CMake, I realised that dune-common dumps all the installation files into the same bucket. While this is practical because one does not have to think what to install and what not, it is much cleaner t...### Description
While modernising my CMake, I realised that dune-common dumps all the installation files into the same bucket. While this is practical because one does not have to think what to install and what not, it is much cleaner to split installation in different components. This way, the user has much more control over the objects that get installed on its system or package. Doing this is simple: Adding a `COMPONENT` argument to every `install` CMake call.
### Proposal
My proposal would be to have 4 components:
| Component | Description | Example
| ------ | ------ | ------ |
| `Runtime` | Final scripts or binaries product of the module | Executables from application modules |
| `Development` | Header files and build system mandatory to continue downstream development | Header files, cmake macros |
| `Library` | Objects needed for both `Development` and `Runtime` usage in downstream modules | libraries for UGGrid or `libdunecommon` |
| `Documentation` | Documentation files | man pages, Doxygen |
In CMake, objects not assigned to any component are set to `Unspecified` (current behaviour). With the proposed setting, nothing should be in this bucket.
### Usage
Assuming components are set, installation of files in a given component is as simple as:
```bash
# User doesn't want to develop, just to link a final application
cmake --install . --component Library
# User wants to develop a downstream module
cmake --install . --component Library --component Development
# User wants to use final binaries
cmake --install . --component Library --component Runtime
# User wants to use everything (current behaviour)
cmake --install .
```
### Is this backwards compatible?
**Yes!** If no component is given on the command line (current usage), all components are installed. This means that the change is backwards compatible with any installation script out there.
### What next?
Discussion... If there seems to be agreement/interest, I can push a MR.CMake Modernizationhttps://gitlab.dune-project.org/core/dune-common/-/issues/245Using dune internal virtual env2023-02-10T15:06:33ZAndreas DednerUsing dune internal virtual envSteps (for the changes see branch `test/use_interal_duneenv`):
- I changed `DunePythonTestCommand.cmake` to use the virtual env python
- I changed `DunePythonInstallPackage.cmake` to use the virtual env python for `install_python`
- I e...Steps (for the changes see branch `test/use_interal_duneenv`):
- I changed `DunePythonTestCommand.cmake` to use the virtual env python
- I changed `DunePythonInstallPackage.cmake` to use the virtual env python for `install_python`
- I enabled the python env and build my dune modules
```
./dune-common/bin/dunecontrol --opts=config.opts all
```
- I then ran
```
./dune-common/build-cmake/run-in-dune-env \
dune-common/bin/setup-dunepy.py --opts=config.opts install
```
This installs the dune packages (editable with my `config.opts` setting) into the dune internal env (due to the use of the `run-in-dune-env` scripts). The correct cmake flags are used by the `setup-dunepy` script. This last step could be done automatically by `dunecontrol all` after configuring all dune modules.
I can now run
```
make test_python
```
in both the build directories of `dune-common` and `dune-grid` and that works fine.
The embedded tests fail at the moment because I'm not sure how to activate the correct python interpreter:
```
make test_embed1
./test_embed1
terminate called after throwing an instance of 'pybind11::error_already_set'
what(): ModuleNotFoundError: No module named 'dune.common'
Aborted (core dumped)
```
while
```
../../../run-in-dune-env ./test_embed1
```
works as expected.https://gitlab.dune-project.org/core/dune-common/-/issues/244use dunecontrol in module.py2023-02-10T15:06:33ZAndreas Dedneruse dunecontrol in module.pyUse `dunecontrol --print` to find all available dune modules and add to `Depends` in `dune.module. And then use
`dunecontrol --only=dune-py all`Use `dunecontrol --print` to find all available dune modules and add to `Depends` in `dune.module. And then use
`dunecontrol --only=dune-py all`https://gitlab.dune-project.org/core/dune-common/-/issues/228Proposal: Single build directory build2023-11-13T12:23:50ZChristoph GrüningerProposal: Single build directory buildWe are still lacking a proper idea how to get rid of the repeated CMake configure runs, which is performed for each module and all its dependencies.
Thus, I want to discuss the idea of a single build directory: the single build director...We are still lacking a proper idea how to get rid of the repeated CMake configure runs, which is performed for each module and all its dependencies.
Thus, I want to discuss the idea of a single build directory: the single build directory build.
1. A single build directory.
2. Within the directory, all core modules are built.
3. CMake configure is run for each module, but only once. All CMake variables, targets, Find modules are available for all modules (without re-run or tricky import).
4. The dependency resolution is done before the built (make) starts. It would be enough to check the presence of a modules `dune.module` file.
5. All build artifacts are co-located, e.g., `lib/` contains libdunecommon, libdunegeometry, libdunegrid and so on.
6. Quasi-circular dependency like dune-geometry and dune-localfunctions are not a problem, because whatever module is built first knows from the dependency resolution whether the other module is present or not and can conditionally built features and tests.
7. As the configuration is done together, we can ensure that all modules have the same flags, the same libraries, the same C++ version and so on.
8. It would be possible to have multiple build flavors (different compilers, flags, external dependencies), one per build directory.
9. Distribution as RPM or Deb would just include the source code, similar to the tarball. The user have to set up a build directory and build libdunecommon, libdunegeometry and so on.CMake Modernization