Core Modules issueshttps://gitlab.dune-project.org/groups/core/-/issues2023-03-01T11:23:16Zhttps://gitlab.dune-project.org/core/dune-common/-/issues/247Overhaul of the Python-CMake integration2023-03-01T11:23:16ZDominic Kempfdominic.kempf@iwr.uni-heidelberg.deOverhaul of the Python-CMake integrationTL;DR: This issue a summarizes the status quo of our Python/CMake integration and proposes some changes. You might want to have a look at the *High Level Goals* part of the proposal below to get an idea.
With the recent discussion of en...TL;DR: This issue a summarizes the status quo of our Python/CMake integration and proposes some changes. You might want to have a look at the *High Level Goals* part of the proposal below to get an idea.
With the recent discussion of enabling the Python bindings by default, our CMake extension for Python code has become more attention. So far, this extension accommodated the needs of several user groups, but did not do this in a unified, coordinated way. With the prospect of default-enabled Python bindings, additional requirements have arisen that require a bit of reengineering of the extension. This issue summarizes all technical aspects and provides a proposal.
## Prerequisistes: Defining *Installation*
This document will talk a lot about *installation*. It is worth to make some fine distinctions here:
* By *Stack Installation* we refer to a method of installing an entire Dune stack including getting all the core sources. An example of this would be Debian packages or Spack repository. We do not cover this type here, because it is not really affected. In the Python context it is important to note that `pip`-installable Dune packages are also in this category and are therefore not subject of this document, although it seems clearly Python-related.
* By *Global Installation* we refer to the installation method that we know from `make install` in C++: A set of sources and build artifacts is copied into a pre-defined directory structure under a standardized system path prefix.
* By *Local Installation* we refer to the same method as *Global installation* only that the user-given prefix that is not a system path.
* By *Package Installation* we refer to the process that is necessary to execute code in a Python environment without manual adjustments to path variables. Although global/local and Package installation of Python code use the same commands (e.g. `python -m pip install`) this still must be distinguished, as it is also part of the build process.
## Prerequisites: Workflow Stages
It is also worth distinguishing the typical stages in a user/developer workflow.
* *CMake stage*: the project is configured.
* *Build stage*: Compiling C++ sources, potentially also building tests
* *Test Stage*: Running tests on the build produced by the build stage
* *Install stage*: Perform global/local installation
## Status Quo:
Currently, the following Python-related things happen at these stages. You can skip over
this part if it is too technical for you, but I feel like there is a big need for a write-up
of this.
* CMake Stage
* Python is searched on the System. It finds the currently active Python,
which might be the System interpreter or a virtualenv that is activated
at the time CMake runs
* We determine whether the found Python is in a virtualenv
* We search for the Python package `pip`
* If the user enabled `DUNE_PYTHON_VIRTUALENV_SETUP`, a virtualenv is created.
This uses either the `virtualenv` or the `venv` package.
If the build directory tree uses an absolute prefix, this env is located in
`build-prefix/dune-python-env`, otherwise it is placed in the build directory
of `dune-common`. If `dune-common` is globally installed, the Dune module
dependency DAG is traversed to find a build directory - this procedure
potentially breaks if there is a diamond dependency pattern with some modules
of the diamond being globally installed. The created virtualenv is only exposed through
the CMake variable `DUNE_PYTHON_VIRTUALENV_EXECUTABLE`, the Python found by CMake
is still accessible. An `activate` script and `run-in-dune-env.sh` wrapper are
placed in each build directory to give users access to the created environment.
* If the virtualenv was set up, Python packages that are provided by this Dune
module are installed into the virtualenv.
* dune-testtools and modules that depend on it continue to execute Python code
during the CMake run.
* Build Stage
* Python Modules that are added with `dune_add_pybind11_module` are built.
* In dune-codegen, some C++ targets depend on a header that is generated running
Python code in the virtualenv. This is managed by the CMake command `add_custom_command`.
* dune-fufem targets link against the Python libraries to start an embedded
interpreter.
* Test Stage
* As part of `ctest`, Python tests that were added through the `dune_python_add_test`
command are executed.
* If you only want to run the Python tests, you can also do `make test_python`.
* When code using the Python bindings is first executed (likely in the test stage),
it sets up a Dune module called `dune-py` that serves as the playground for code
generation and Just-In-Time compilation. This module is currently not placed in
an isolated per-build location leading to potential compatibility issues between builds.
* Install Stage
* A global/local installation of Python packages is performed via `pip`. In contrast to C++,
the prefix choice depends on the CMake Variable `DUNE_PYTHON_INSTALL_LOCATION=user|system|none`,
where
* `none` disables installation of Python packages
* `system` uses the interpreter found by CMake
* `user` uses `pip install --user`, which installs into `~/.local`
If the variable is not set, it defaults to `system` if the found interpreter
runs in a virtualenv and `none` otherwise. The `user` option is incompatible
with virtual environments.
* A wheel (a standarized binary distribution format for Python packages) is built
for each package and installed into a location under the Dune installation prefix (called a wheelhouse).
This is necessary to allow partially installed Dune stacks: If my module depends
on dune-common - where do I get the Python package `dune.common` from? Answer: from
the Dune wheelhouse under the prefix that was used to install dune-common.
* Dependencies of Python packages are automatically installed by pip using the
Python Package Index: This requires network access. Failing to provide it currently
results in a CMake Error (pip network timeout).
* The Python-specific part of the installation can be run separately by using
the dedicated `make install_python`. This is what is advocated and used for
the Python bindings - although you need to do this before you can run the test
stage.
* All installation rules refer to the Python environment that was active when CMake ran - which
might be different from the Python environment at the time of installation.
If you want to read code for this, it is located in `dune-common/cmake/modules/DunePython*.cmake`.
The main entry point is `DunePythonCommonMacros.cmake`.
## Proposal
I am willing to implement this or a similar solution.
### High Level Goals
* With Dune being and staying a C++ project, the above workflow stages and their order should stay the same.
* *At the build stage, all Python code is package-installed into an isolated per-build environment.* (This was a fundamental outcome of the meeting: In the C++ frame of mind, Python Package Installation is part of building, not installing).
* Building a Dune stack without network access should be successful, though Python parts can be disabled in this build.
* Global/local installation of Python should install all Python packages into the Python environment that was active when CMake ran
* ~~Global/local installation of Python should depend on the Python environment at the time of installation~~ (I really wanted this to work, but thinking about it more I realized that the embedded interpreter case struggles heavily with this)
* Performing a global/local installation of the entire Dune stack (e.g. `dunecontrol make install_python`)
should make the build environment disposable.
* Running Python code should be independent of Dune-specific environment variables. (I added this one after some experiments)
### Technical Details
I think, we already have almost all of the necessary components, they just need to be slightly tweaked.
These are only some core ideas, there is a lot of detail to this that can be discussed
once I provided a sample implementation:
* The virtualenv set up is enabled by default
* The following conditions will lead to all Python aspects being disabled
* Python is not found
* Pip or virtualenv/venv are not found
* Pip operations produce a network timeout (this can still be worked around
by setting up PyPI mirrors, but this should never be part of a standard user workflow)
* After virtualenv set up, the CMake check for Python is performed again, so that
CMake only knows the virtual environment.
* Python Package installations into the virtualenv are formulated as custom targets
that are part of `all` iff the bindings are enabled.
* ~~Package installation is executed in CMake Script Mode (`cmake -P`) to allow searching
and finding the current Python environment in the installation stage.~~
* The embedded interpreter use case should use CMake generator expressions to link
to the system Python after local/global installation.
The only problematic use case that I can think of right now is the dune-testtools
one - but that mostly affects myself. I will find a way to deal with it.https://gitlab.dune-project.org/core/dune-common/-/issues/246Use CMake components when installing2023-11-13T12:12:16ZSantiago Ospina De Los Ríossospinar@gmail.comUse CMake components when installing### Description
While modernising my CMake, I realised that dune-common dumps all the installation files into the same bucket. While this is practical because one does not have to think what to install and what not, it is much cleaner t...### Description
While modernising my CMake, I realised that dune-common dumps all the installation files into the same bucket. While this is practical because one does not have to think what to install and what not, it is much cleaner to split installation in different components. This way, the user has much more control over the objects that get installed on its system or package. Doing this is simple: Adding a `COMPONENT` argument to every `install` CMake call.
### Proposal
My proposal would be to have 4 components:
| Component | Description | Example
| ------ | ------ | ------ |
| `Runtime` | Final scripts or binaries product of the module | Executables from application modules |
| `Development` | Header files and build system mandatory to continue downstream development | Header files, cmake macros |
| `Library` | Objects needed for both `Development` and `Runtime` usage in downstream modules | libraries for UGGrid or `libdunecommon` |
| `Documentation` | Documentation files | man pages, Doxygen |
In CMake, objects not assigned to any component are set to `Unspecified` (current behaviour). With the proposed setting, nothing should be in this bucket.
### Usage
Assuming components are set, installation of files in a given component is as simple as:
```bash
# User doesn't want to develop, just to link a final application
cmake --install . --component Library
# User wants to develop a downstream module
cmake --install . --component Library --component Development
# User wants to use final binaries
cmake --install . --component Library --component Runtime
# User wants to use everything (current behaviour)
cmake --install .
```
### Is this backwards compatible?
**Yes!** If no component is given on the command line (current usage), all components are installed. This means that the change is backwards compatible with any installation script out there.
### What next?
Discussion... If there seems to be agreement/interest, I can push a MR.CMake Modernizationhttps://gitlab.dune-project.org/core/dune-common/-/issues/245Using dune internal virtual env2023-02-10T15:06:33ZAndreas DednerUsing dune internal virtual envSteps (for the changes see branch `test/use_interal_duneenv`):
- I changed `DunePythonTestCommand.cmake` to use the virtual env python
- I changed `DunePythonInstallPackage.cmake` to use the virtual env python for `install_python`
- I e...Steps (for the changes see branch `test/use_interal_duneenv`):
- I changed `DunePythonTestCommand.cmake` to use the virtual env python
- I changed `DunePythonInstallPackage.cmake` to use the virtual env python for `install_python`
- I enabled the python env and build my dune modules
```
./dune-common/bin/dunecontrol --opts=config.opts all
```
- I then ran
```
./dune-common/build-cmake/run-in-dune-env \
dune-common/bin/setup-dunepy.py --opts=config.opts install
```
This installs the dune packages (editable with my `config.opts` setting) into the dune internal env (due to the use of the `run-in-dune-env` scripts). The correct cmake flags are used by the `setup-dunepy` script. This last step could be done automatically by `dunecontrol all` after configuring all dune modules.
I can now run
```
make test_python
```
in both the build directories of `dune-common` and `dune-grid` and that works fine.
The embedded tests fail at the moment because I'm not sure how to activate the correct python interpreter:
```
make test_embed1
./test_embed1
terminate called after throwing an instance of 'pybind11::error_already_set'
what(): ModuleNotFoundError: No module named 'dune.common'
Aborted (core dumped)
```
while
```
../../../run-in-dune-env ./test_embed1
```
works as expected.https://gitlab.dune-project.org/core/dune-common/-/issues/244use dunecontrol in module.py2023-02-10T15:06:33ZAndreas Dedneruse dunecontrol in module.pyUse `dunecontrol --print` to find all available dune modules and add to `Depends` in `dune.module. And then use
`dunecontrol --only=dune-py all`Use `dunecontrol --print` to find all available dune modules and add to `Depends` in `dune.module. And then use
`dunecontrol --only=dune-py all`https://gitlab.dune-project.org/core/dune-common/-/issues/228Proposal: Single build directory build2023-11-13T12:23:50ZChristoph GrüningerProposal: Single build directory buildWe are still lacking a proper idea how to get rid of the repeated CMake configure runs, which is performed for each module and all its dependencies.
Thus, I want to discuss the idea of a single build directory: the single build director...We are still lacking a proper idea how to get rid of the repeated CMake configure runs, which is performed for each module and all its dependencies.
Thus, I want to discuss the idea of a single build directory: the single build directory build.
1. A single build directory.
2. Within the directory, all core modules are built.
3. CMake configure is run for each module, but only once. All CMake variables, targets, Find modules are available for all modules (without re-run or tricky import).
4. The dependency resolution is done before the built (make) starts. It would be enough to check the presence of a modules `dune.module` file.
5. All build artifacts are co-located, e.g., `lib/` contains libdunecommon, libdunegeometry, libdunegrid and so on.
6. Quasi-circular dependency like dune-geometry and dune-localfunctions are not a problem, because whatever module is built first knows from the dependency resolution whether the other module is present or not and can conditionally built features and tests.
7. As the configuration is done together, we can ensure that all modules have the same flags, the same libraries, the same C++ version and so on.
8. It would be possible to have multiple build flavors (different compilers, flags, external dependencies), one per build directory.
9. Distribution as RPM or Deb would just include the source code, similar to the tarball. The user have to set up a build directory and build libdunecommon, libdunegeometry and so on.CMake Modernizationhttps://gitlab.dune-project.org/core/dune-common/-/issues/227Build wheels for mac/linux2023-02-10T15:06:33ZAndreas DednerBuild wheels for mac/linuxPossibly could be done through CI using https://github.com/joerick/cibuildwheel
An example combined with pybind11: https://github.com/joerick/cibuildwheel
Also build conda packages, e.g., https://github.com/pybind/python_examplePossibly could be done through CI using https://github.com/joerick/cibuildwheel
An example combined with pybind11: https://github.com/joerick/cibuildwheel
Also build conda packages, e.g., https://github.com/pybind/python_examplehttps://gitlab.dune-project.org/core/dune-common/-/issues/226handling of python bindings by dune-project and dunecontrol2023-03-01T11:23:16ZAndreas Dednerhandling of python bindings by dune-project and dunecontrol# `duneproject`
- no support for adding python bindings to a new project.
- for the pip installation we have added some new (optional) fields to `dune.module` (see !900) which would be nice to have in new modules as well.
- In the list...# `duneproject`
- no support for adding python bindings to a new project.
- for the pip installation we have added some new (optional) fields to `dune.module` (see !900) which would be nice to have in new modules as well.
- In the list of found dune modules it would be good to disregard a possibly found `dune-py` since no module should depend on that.
- A `README` file is generated but perhaps that should be `README.md`?
# `dunecontrol`: fixed
- if `dune-py` is found `dunecontrol` tries to build it but fails (it's in source build). This module should be ignored by `dunecontrol` - possibly this needs to be done in `dunemodules.lib`https://gitlab.dune-project.org/core/dune-common/-/issues/224Handling of circular dependencies2024-02-22T11:18:25ZChristian EngwerHandling of circular dependenciesWe had a couple of cases where we came to the conclusion that these would impeed a circular dependency between dune modules.
### Example
One example would be higher order geometries in `dune-geometry`. The canonical way to implement the...We had a couple of cases where we came to the conclusion that these would impeed a circular dependency between dune modules.
### Example
One example would be higher order geometries in `dune-geometry`. The canonical way to implement these should be based on `dune-localfunctions`. A geometry describes a particular type of function in local coordinates. The issue is that `dune-localfunctions` already depends on `dune-geometry`. This would (potentially) lead to the following setup:
```
Module: dune-localfunctions
Depends: dune-geometry
```
and
```
Module: dune-geometry
Depends: dune-common
Suggests: dune-localfunctions
```
### Current state
The current state is, that his might even work, although not officially supported.
`dunecontrol` simply starts building a graph of all dependencies and suggesttions and when ever a module is already in the list, it will not add it again. The only difference between `Suggests` and `Depends` is that missing dependencies are fatal, while missing suggestions are OK.
So depending on the order in which `dunecontrol` finds the modules, it might create one of two possible DAGS of the core modules (as an example). Here follow the original graph and the two possible DAGS and their respective build-order:
```mermaid
graph RL
subgraph AA[Original Graph]
A:common(dune-common)
A:geometry(dune-geometry)
A:istl(dune-istl)
A:localfunctions(dune-localfunctions)
A:grid(dune-grid)
A:geometry --> A:common
A:istl --> A:common
A:geometry -.-> A:localfunctions
A:localfunctions --> A:geometry
A:localfunctions --> A:common
A:grid --> A:geometry
end
```
```mermaid
graph RL
subgraph D1[DAG variant 1]
B:common(dune-common)
B:geometry(dune-geometry)
B:istl(dune-istl)
B:localfunctions(dune-localfunctions)
B:grid(dune-grid)
B:geometry --> B:common
B:istl --> B:common
B:localfunctions --> B:geometry
B:localfunctions --> B:common
B:grid --> B:geometry
end
```
```mermaid
graph RL
subgraph D2[DAG variant 2]
C:common(dune-common)
C:geometry(dune-geometry)
C:istl(dune-istl)
C:localfunctions(dune-localfunctions)
C:grid(dune-grid)
C:geometry --> C:common
C:localfunctions --> C:common
C:istl --> C:common
C:grid --> C:geometry
C:geometry -.-> C:localfunctions
end
```
```mermaid
graph LR
subgraph B1[Build order 1]
D:common(dune-common)
D:geometry(dune-geometry)
D:istl(dune-istl)
D:localfunctions(dune-localfunctions)
D:grid(dune-grid)
D:common --> D:istl
D:istl --> D:geometry
D:grid --> D:localfunctions
D:geometry --> D:grid
end
```
```mermaid
graph LR
subgraph B2[Build order 2]
E:common(dune-common)
E:geometry(dune-geometry)
E:istl(dune-istl)
E:localfunctions(dune-localfunctions)
E:grid(dune-grid)
E:common --> E:istl
E:istl --> E:localfunctions
E:localfunctions -.-> E:geometry
E:geometry --> E:grid
end
```
As the circular dependency is only *weak* in the sense, that `dune-localfunctions` only requires the geometry headers but no lib, it is possible to compile this setup without problems.
**Note** that there might still be some issues with `cmake` though...
### Proposal
In order to fix this problem and be sufficiently flexible to handle such *weak* suggestions and *weak* dependencies I suggest to introduce two additional key words `Requires` and `Recommends`.
The semantics are then as follows:
* `Depends`: requirements that need to be built first (e.g. we might require libs built in these modules).
* `Suggests`: same as `depends`, but we are still functional without these modules.
* `Requires`: we only require the headers, but these relations don't impact the construction of a DAG (in particular we don't require any libs or other compiled artifacts from these modules).
* `Recommends`: same as `Requires`, but we are still functional without these modules.
Alternative names might also be `Weak-Depends` and `Weak-Suggests`... in general I'm open for suggestions.https://gitlab.dune-project.org/core/dune-common/-/issues/221Future direction of dunecontrol2021-06-25T22:01:09ZChristoph GrüningerFuture direction of dunecontrolIn #200 a discussion about the future of dunecontrol started. I want to move the discussion here for better documentation and referencing.
There are two main directions, which I sense:
- Add features to dunecontrol to help users. The ar...In #200 a discussion about the future of dunecontrol started. I want to move the discussion here for better documentation and referencing.
There are two main directions, which I sense:
- Add features to dunecontrol to help users. The are handy to use and easy to teach.
- Try to reduce dunecontrol and make the use more compatible to to the way Compilers, CMake and Make are used in the world outside of Dune. Then users can more easily use the advice they find on Stackoverflow or what they are used from other projects.https://gitlab.dune-project.org/core/dune-istl/-/issues/96Policy/Assumptions on field_types in MultitypeBlockVector/Matrix?2020-10-07T18:36:25ZMarkus BlattPolicy/Assumptions on field_types in MultitypeBlockVector/Matrix?I am trying to get rid of some warnings about shadowing variables/using
directions. Some of them are due to using field_type = ... being reset
in some methods (e.g. norms). I don't want to break anything, hence the
question arises:
Do w...I am trying to get rid of some warnings about shadowing variables/using
directions. Some of them are due to using field_type = ... being reset
in some methods (e.g. norms). I don't want to break anything, hence the
question arises:
Do we insist on field_type being the same for each entry? Seems like the
code at least insists that the field_types of all entries are convertible
to the one of the very first entry.
For the template parameters, arbitrary different types are possible.
Our tests assume always use the same type.https://gitlab.dune-project.org/core/dune-common/-/issues/220SIMD ABI change warnings (g++-8)2024-02-22T11:19:31ZMarkus BlattSIMD ABI change warnings (g++-8)```
dune-common/dune/common/simd/loop.hh: In function ‘auto Dune::Simd::Overloads::cond(Dune::Simd::Overloads::ADLTag<5, (std::is_same<bool, typename Dune::Simd::Overloads::ScalarType<typename std::decay<_Tp>::type>::type>::value && (lan...```
dune-common/dune/common/simd/loop.hh: In function ‘auto Dune::Simd::Overloads::cond(Dune::Simd::Overloads::ADLTag<5, (std::is_same<bool, typename Dune::Simd::Overloads::ScalarType<typename std::decay<_Tp>::type>::type>::value && (lanes<M>() == lanes<Dune::LoopSIMD<T, S, A> >()))>, M, Dune::LoopSIMD<T, S, A>, Dune::LoopSIMD<T, S, A>) [with M = Dune::LoopSIMD<bool, 5, 64>; T = long double; long unsigned int S = 5; long unsigned int A = 64]’:
dune-common/dune/common/simd/loop.hh:346:12: note: The ABI for passing parameters with 64-byte alignment has changed in GCC 4.6
auto cond(ADLTag<5, std::is_same<bool, Simd::Scalar<M> >::value
^~~~
```https://gitlab.dune-project.org/core/dune-grid/-/issues/117Heap-Buffer-Overflow in gmshtest-onedgrid2021-07-04T11:25:06ZTobias LeibnerHeap-Buffer-Overflow in gmshtest-onedgridWhen compiling the tests with clang's ``-fsanitize=address -fsanitize=undefined``, apart from [several warnings about undefined behavior introduced by dune-uggrid](https://gitlab.dune-project.org/staging/dune-uggrid/-/issues/54), the fol...When compiling the tests with clang's ``-fsanitize=address -fsanitize=undefined``, apart from [several warnings about undefined behavior introduced by dune-uggrid](https://gitlab.dune-project.org/staging/dune-uggrid/-/issues/54), the following error shows up:
```
13: Test command: build/clang-sanitizer/dune-grid/dune/grid/io/file/test/gmshtest-onedgrid
13: Test timeout computed to be: 300
13: Using OneDGrid
13: Reading mesh file dune-grid/doc/grids/gmsh/oned-testgrid.msh
13: Reading 1d Gmsh grid...
13: version 2.1 Gmsh file detected
13: file contains 10 nodes
13: file contains 11 elements
13: number of real vertices = 10
13: number of boundary elements = 2
13: number of elements = 9
13: 2, 0, 0
13: 2, 1, 2
13: =================================================================
13: ==1464084==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60200003b5b8 at pc 0x00000051cb03 bp 0x7ffd492a2b30 sp 0x7ffd492a2b28
13: READ of size 4 at 0x60200003b5b8 thread T0
13: #0 0x51cb02 in void testReadingAndWritingGrid<Dune::OneDGrid>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, bool) dune-grid/dune/grid/io/file/test/gmshtest.cc:85:56
13: #1 0x5136be in main dune-grid/dune/grid/io/file/test/gmshtest.cc:298:3
13: #2 0x7feaf63f80b2 in __libc_start_main /build/glibc-YYA7BZ/glibc-2.31/csu/../csu/libc-start.c:308:16
13: #3 0x4689dd in _start (build/clang-sanitizer/dune-grid/dune/grid/io/file/test/gmshtest-onedgrid+0x4689dd)
13:
13: 0x60200003b5b8 is located 0 bytes to the right of 8-byte region [0x60200003b5b0,0x60200003b5b8)
13: allocated by thread T0 here:
13: #0 0x51088d in operator new(unsigned long) (build/clang-sanitizer/dune-grid/dune/grid/io/file/test/gmshtest-onedgrid+0x51088d)
13: #1 0x548b29 in __gnu_cxx::new_allocator<int>::allocate(unsigned long, void const*) /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/ext/new_allocator.h:114:27
13: #2 0x548a8d in std::allocator_traits<std::allocator<int> >::allocate(std::allocator<int>&, unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/alloc_traits.h:444:20
13: #3 0x548a13 in std::_Vector_base<int, std::allocator<int> >::_M_allocate(unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_vector.h:343:20
13: #4 0x55afd3 in std::vector<int, std::allocator<int> >::_M_default_append(unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/vector.tcc:635:34
13: #5 0x557d71 in std::vector<int, std::allocator<int> >::resize(unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_vector.h:937:4
13: #6 0x53c7a7 in Dune::GmshReaderParser<Dune::OneDGrid>::read(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) dune-grid/dune/grid/io/file/gmshreader.hh:472:38
13: #7 0x532a89 in Dune::GmshReader<Dune::OneDGrid>::do_read(Dune::GridFactory<Dune::OneDGrid>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<int, std::allocator<int> >&, std::vector<int, std::allocator<int> >&, bool, bool) dune-grid/dune/grid/io/file/gmshreader.hh:805:16
13: #8 0x5231b2 in Dune::GmshReader<Dune::OneDGrid>::read(Dune::GridFactory<Dune::OneDGrid>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Dune::GmshReader<Dune::OneDGrid>::DataFlagArg, Dune::GmshReader<Dune::OneDGrid>::DataArg, bool) dune-grid/dune/grid/io/file/gmshreader.hh:938:7
13: #9 0x51bc23 in void testReadingAndWritingGrid<Dune::OneDGrid>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, bool) dune-grid/dune/grid/io/file/test/gmshtest.cc:73:3
13: #10 0x5136be in main dune-grid/dune/grid/io/file/test/gmshtest.cc:298:3
13: #11 0x7feaf63f80b2 in __libc_start_main /build/glibc-YYA7BZ/glibc-2.31/csu/../csu/libc-start.c:308:16
13:
13: SUMMARY: AddressSanitizer: heap-buffer-overflow dune-grid/dune/grid/io/file/test/gmshtest.cc:85:56 in void testReadingAndWritingGrid<Dune::OneDGrid>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, bool)
13: Shadow bytes around the buggy address:
13: 0x0c047ffff660: fa fa 06 fa fa fa 06 fa fa fa 06 fa fa fa 06 fa
13: 0x0c047ffff670: fa fa 06 fa fa fa 00 fa fa fa fd fd fa fa fd fa
13: 0x0c047ffff680: fa fa 04 fa fa fa 04 fa fa fa 04 fa fa fa 00 fa
13: 0x0c047ffff690: fa fa 05 fa fa fa 03 fa fa fa fd fa fa fa 00 fa
13: 0x0c047ffff6a0: fa fa 07 fa fa fa 04 fa fa fa 00 fa fa fa 00 00
13: =>0x0c047ffff6b0: fa fa 00 07 fa fa 00[fa]fa fa fd fa fa fa fd fa
13: 0x0c047ffff6c0: fa fa fd fa fa fa 00 fa fa fa fd fa fa fa fd fa
13: 0x0c047ffff6d0: fa fa fd fa fa fa fd fd fa fa fd fa fa fa fd fa
13: 0x0c047ffff6e0: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa
13: 0x0c047ffff6f0: fa fa fd fa fa fa 00 fa fa fa 00 fa fa fa 00 fa
13: 0x0c047ffff700: fa fa 00 fa fa fa 00 fa fa fa 00 fa fa fa fa fa
13: Shadow byte legend (one shadow byte represents 8 application bytes):
13: Addressable: 00
13: Partially addressable: 01 02 03 04 05 06 07
13: Heap left redzone: fa
13: Freed heap region: fd
13: Stack left redzone: f1
13: Stack mid redzone: f2
13: Stack right redzone: f3
13: Stack after return: f5
13: Stack use after scope: f8
13: Global redzone: f9
13: Global init order: f6
13: Poisoned by user: f7
13: Container overflow: fc
13: Array cookie: ac
13: Intra object redzone: bb
13: ASan internal: fe
13: Left alloca redzone: ca
13: Right alloca redzone: cb
13: Shadow gap: cc
13: ==1464084==ABORTING
1/1 Test #13: gmshtest-onedgrid ................***Failed 0.60 sec
```
The problem occurs in https://gitlab.dune-project.org/core/dune-grid/-/blob/master/dune/grid/io/file/test/gmshtest.cc#L84, where ``gridFactory.insertionIndex(intersection)`` becomes ``2`` although ``boundaryIDs.size() == 2``.https://gitlab.dune-project.org/core/dune-grid/-/issues/114Tautological compare in GeometryGrid2020-08-13T12:33:26ZChristoph GrüningerTautological compare in GeometryGridI tested some of Oliver's code with dune-grid 2.7 and Clang 10 complained the following:
```
/home/gruech/dune/2.7/dune-grid/dune/grid/geometrygrid/geometry.hh:34:87: warning: bitwise comparison always evaluates to false [-Wtautological-...I tested some of Oliver's code with dune-grid 2.7 and Clang 10 complained the following:
```
/home/gruech/dune/2.7/dune-grid/dune/grid/geometrygrid/geometry.hh:34:87: warning: bitwise comparison always evaluates to false [-Wtautological-compare]
34 | static const bool v = hasSingleGeometryType::v && ((mydim == dim) || ((id | 1u) == 1u) || ((id | 1u) == idMask));
| ~~~~~~~~~~~^~~~~~
```
Quite interesting, but I don't know what should be the actual statement. Can someone help?https://gitlab.dune-project.org/core/dune-istl/-/issues/94METIS vs scotchmetis and ParMETIS vs ptscotchparmetis2020-10-28T16:07:51ZSimon PraetoriusMETIS vs scotchmetis and ParMETIS vs ptscotchparmetis### Summary
The file `repartition.hh` is the only place in the core modules using (Scotch)METIS or (Scotch)ParMETIS. There are currently some discussions in https://gitlab.dune-project.org/core/dune-common/-/merge_requests/822 and https:...### Summary
The file `repartition.hh` is the only place in the core modules using (Scotch)METIS or (Scotch)ParMETIS. There are currently some discussions in https://gitlab.dune-project.org/core/dune-common/-/merge_requests/822 and https://gitlab.dune-project.org/core/dune-common/-/merge_requests/821 whether the scotch variants should be dropped. This issues should provide some insights into this problem and tries to find a solution.
### Details
- Scotch implements the interface of some (Par)METIS functions, like `ParMETIS_V3_PartKway`. It ships under the open-source license CeCILL-C (compatible to GPL(?))
- METIS is under the Apache License Version 2.0, while ParMETIS is distributed under a special license for educational, research, non-profit and US-agency use only.
- debian-free does not ship parmetis
- The interface of Scotch is compatible with METIS-4/5, while the interface of PTScotch seems to be compatible with ParMETIS-4
- ParMETIS, on the other hand, requires METIS >= 5
- METIS >= 5 introduces typedefs for its real and index types, see discussion in https://gitlab.dune-project.org/core/dune-istl/issues/45
In summary: One can either use METIS-5 + ParMETIS-4, or Scotch + PTScotch. Not a mixup of scotch and METIS.
Scotch does not (in no version) provide the typedefs of METIS-5, but it is provided in several independent version for index types that makes it hard to configure.
Some more problems:
- Scotch-METIS is not fully compatible to METIS. In recent scotch version >= 6.0.7 you can select which METIS API you want to provide. Either API v3 or API v5. Both are not exactly the same as in METIS, but at least similar. Some function arguments even have a different meaning and the same values would result in completely different behaviour.
- PTScotch-ParMETIS is compatible to ParMETIS. There is only one API.
- PTScotch does not "export-C" its functions. This is a problem for using the library in C++ and introduces some restrictions on dependent libraries like MPI that, because of this, need to be provided for C and not for C++ plus lots of required platform dependent definitions
### Discussion
1. Do we want to support the (pt)scotch interface at all?
2. Do we want to assume that when calling `find_package(METIS)` in cmake we silently can also get scotch (and correspondingly for ParMETIS and PTScotch)?
3. Currently there is a workaround for the missing typedefs implemented in `dune/istl/repartition.hh` only, see MR https://gitlab.dune-project.org/core/dune-istl/-/merge_requests/173 Do we want to provide a similar workaround for the general core modules? This seems reasonable, since the find package is provided with dune-common.
4. Is there experience with scotch and ptscotch? Can someone test the changes in the cmake modules in https://gitlab.dune-project.org/core/dune-common/-/merge_requests/822 and https://gitlab.dune-project.org/core/dune-common/-/merge_requests/821https://gitlab.dune-project.org/core/dune-common/-/issues/212Headercheck??2021-03-22T16:27:33ZSimon PraetoriusHeadercheck??What is the "headercheck" feature about? In the comments it just says: "reimplementation of the old autotools feaure". It seems to be disabled by default.
- Does anyone use this feature?
- How does it work and what does it do? It is n...What is the "headercheck" feature about? In the comments it just says: "reimplementation of the old autotools feaure". It seems to be disabled by default.
- Does anyone use this feature?
- How does it work and what does it do? It is not really documented!
- There are some functions called to remove headers from headercheck. Why is this necessary?
- Is it a necessary "feature" or just a historic inheritance from the autotools build-system?
- Is there a modern alternative?
Guessing wildly, by interpreting the name "headercheck", I would say this feature somehow checks that header files are (syntactically) valid. Since we have a lot of header-only code, that would not be tested otherwise, this could be useful. If this is the intend of that feature, we could replace it with precompiled headers. This would additionally give a spead-up in the compilation time of about 20%. CMake provides simple functionality for this purpose. See https://gitlab.dune-project.org/simon.praetorius/dune-cmake/issues/3 for a discussion of this feature.New CMake Build-Systemhttps://gitlab.dune-project.org/core/dune-common/-/issues/207test failure on i386 (simd stuff)2020-05-22T18:07:26ZAnsgar Burchardtansgar.burchardt@tu-dresden.detest failure on i386 (simd stuff)Some tests fail on i386 (without SSE), for example:
```
/<<PKGBUILDDIR>>/dune/common/simd/test.hh:1100: In checkBinaryOpVS, while testing Dune::Simd::UnitTest::OpInfixDiv (Dune::AlignedNumberImpl::AlignedNumber<double, 32u>&, double&): E...Some tests fail on i386 (without SSE), for example:
```
/<<PKGBUILDDIR>>/dune/common/simd/test.hh:1100: In checkBinaryOpVS, while testing Dune::Simd::UnitTest::OpInfixDiv (Dune::AlignedNumberImpl::AlignedNumber<double, 32u>&, double&): Error: check (lane(l, vopres) == static_cast<T>(op(lane(l, static_cast<V1>(vref1)), static_cast<T2>(sref2) ))) failed
```
See also the [full build log](https://buildd.debian.org/status/fetch.php?pkg=dune-common&arch=i386&ver=2.7.0-1&stamp=1590056277&raw=0)
Adding `-ffloat-store` to the compiler flags works around the problem, so one side of the comparison probably wasn't stored to memory and has still extra precision in the register. I'll do that for the Debian packages for now.
@joe wrote the tests, but I'm not sure how much time to invest in i386. I don't currently have time to spend on this. !387 fixed similar issues (and the number of the merge request is probably the best part of this!)https://gitlab.dune-project.org/core/dune-grid/-/issues/110Gridinfo executable does not build2020-05-14T20:31:06ZDominic Kempfdominic.kempf@iwr.uni-heidelberg.deGridinfo executable does not buildIn almost 10 years of Dune, it did not know these executables exist. This is partly because you have to enable them by setting `ENABLE_GRID_EXTRA_UTILS=1`, but the `DuneGridMacros` file claims that the variable is called `DUNE_GRID_EXTRA...In almost 10 years of Dune, it did not know these executables exist. This is partly because you have to enable them by setting `ENABLE_GRID_EXTRA_UTILS=1`, but the `DuneGridMacros` file claims that the variable is called `DUNE_GRID_EXTRA_UTILS`. If I enable it, the UG variant fails with this error:
```
/home/domse/structures/dune-grid/dune/grid/utility/gridinfo.hh:184:28: error: ‘const class Dune::MultipleCodimMultipleGeomTypeMapper<Dune::GridView<Dune::UGGridLeafGridViewTraits<const Dune::UGGrid<3> > > >’ has no member named ‘map’
184 | int index = mapper.map(e, i, codim);
```
I do not want to know what happens if I try the Alberta one.
I guess the compilation error should be fixed and the variable name should be corrected.https://gitlab.dune-project.org/core/dune-common/-/issues/198Status of TBB2023-09-20T07:17:05ZNils-Arne DreierStatus of TBBI'm planing to use TBB for thread parallelism in a ISTL based project. I'm willing to prepare a MR in dune-istl when I introduced all the TBB calls. But I'm wondering what is the actual status of TBB in dune?
The build infrastructure is ...I'm planing to use TBB for thread parallelism in a ISTL based project. I'm willing to prepare a MR in dune-istl when I introduced all the TBB calls. But I'm wondering what is the actual status of TBB in dune?
The build infrastructure is there but it is not used in the core modules.
On the 2017 developer meeting was a discussion about that: https://dune-project.org/community/meetings/2017-03-devmeeting/threading-infrastructure/
That is what was agreed on:
> We could introduce an abstraction layer. But that only makes sense if we have at least a rough idea regarding the possible variability of the interface. We do not know about any other possible interface. And wrapping will in some cases be quite expensive (support wise)
and
>We standardise on TBB without abstraction/wrappers. TBB remains optional: if not available there will be not automatic threading in the core modules. Infrastructure that only makes sense in connection with TBB may be missing then. - We do not allow OpenMP in core modules. Users may use OpenMP in their own modules. - We generally disallow threads not managed by the thread pool. In special cases you may start threads outside (e.g. for communicating in the background), but you need to ensure that there are no adverse effects, e.g. performace loss due to oversubscription. - When there is equivalent functionality in TBB and the C++ standard, prefer the standard functionality.
Does that mean, that whenever I want to use TBB in the core modules if have to provide a fallback like
```c++
#if HAVE_TBB
tbb::parallel_for(..., [](...){
...
}):
#else
for(...){
...
}
#endif
```
I think that would lead to very ugly code. Should we provide at least convenient functions for the most used cases like `parallel_for`?https://gitlab.dune-project.org/core/dune-istl/-/issues/90NonoverlappingSchwarzOperator doesn't work for MultiTypeBlockMatrix2020-04-06T08:39:42ZTimo KochNonoverlappingSchwarzOperator doesn't work for MultiTypeBlockMatrixCurrent implementation in `novlschwarz.hh` doesn't compile and work for MultiTypeBlockMatrices as `matrix_type`.Current implementation in `novlschwarz.hh` doesn't compile and work for MultiTypeBlockMatrices as `matrix_type`.https://gitlab.dune-project.org/core/dune-grid/-/issues/106Lifetime guarantees for geometry objects obtained through the grid interface2022-01-09T15:35:12ZTimo KochLifetime guarantees for geometry objects obtained through the grid interfaceI found this very surprising many times before, so I thought I'll open an issue.
I understand I only get "wrappers" from the grid when I call `element.geometry()` or `intersection.geometry()` in
a loop over the grid.
```cpp
for (const ...I found this very surprising many times before, so I thought I'll open an issue.
I understand I only get "wrappers" from the grid when I call `element.geometry()` or `intersection.geometry()` in
a loop over the grid.
```cpp
for (const auto& element : elements(leafGridView))
for (const auto& intersection : intersections(leafGridView, element))
...
```
Still a really (in my opinion) natural thing to do would be to store some of the geometries and do something with them later. In this sense I wanna use them as real geometry objects like the name suggests. Meaning I want value semantics (if I'm using that right). But unfortunately, e.g. for ALUGrid, I only get pointers to some internally cached version that changes when I go to the next element.
Is there a way to achieve this (without manually constructing a copy of the geometry by extracting type and corners) that I'm missing? Do such guarantees exist since 2.6 for other objects like entities?
This also seems like a problem for thread safety if there are internal mutable caches. But I guess that could be solved without too much interface restrictions by each grid implementation by locking internally?