dune-common issueshttps://gitlab.dune-project.org/core/dune-common/-/issues2023-11-13T12:23:50Zhttps://gitlab.dune-project.org/core/dune-common/-/issues/228Proposal: Single build directory build2023-11-13T12:23:50ZChristoph GrüningerProposal: Single build directory buildWe are still lacking a proper idea how to get rid of the repeated CMake configure runs, which is performed for each module and all its dependencies.
Thus, I want to discuss the idea of a single build directory: the single build director...We are still lacking a proper idea how to get rid of the repeated CMake configure runs, which is performed for each module and all its dependencies.
Thus, I want to discuss the idea of a single build directory: the single build directory build.
1. A single build directory.
2. Within the directory, all core modules are built.
3. CMake configure is run for each module, but only once. All CMake variables, targets, Find modules are available for all modules (without re-run or tricky import).
4. The dependency resolution is done before the built (make) starts. It would be enough to check the presence of a modules `dune.module` file.
5. All build artifacts are co-located, e.g., `lib/` contains libdunecommon, libdunegeometry, libdunegrid and so on.
6. Quasi-circular dependency like dune-geometry and dune-localfunctions are not a problem, because whatever module is built first knows from the dependency resolution whether the other module is present or not and can conditionally built features and tests.
7. As the configuration is done together, we can ensure that all modules have the same flags, the same libraries, the same C++ version and so on.
8. It would be possible to have multiple build flavors (different compilers, flags, external dependencies), one per build directory.
9. Distribution as RPM or Deb would just include the source code, similar to the tarball. The user have to set up a build directory and build libdunecommon, libdunegeometry and so on.CMake Modernizationhttps://gitlab.dune-project.org/core/dune-common/-/issues/246Use CMake components when installing2023-11-13T12:12:16ZSantiago Ospina De Los Ríossospinar@gmail.comUse CMake components when installing### Description
While modernising my CMake, I realised that dune-common dumps all the installation files into the same bucket. While this is practical because one does not have to think what to install and what not, it is much cleaner t...### Description
While modernising my CMake, I realised that dune-common dumps all the installation files into the same bucket. While this is practical because one does not have to think what to install and what not, it is much cleaner to split installation in different components. This way, the user has much more control over the objects that get installed on its system or package. Doing this is simple: Adding a `COMPONENT` argument to every `install` CMake call.
### Proposal
My proposal would be to have 4 components:
| Component | Description | Example
| ------ | ------ | ------ |
| `Runtime` | Final scripts or binaries product of the module | Executables from application modules |
| `Development` | Header files and build system mandatory to continue downstream development | Header files, cmake macros |
| `Library` | Objects needed for both `Development` and `Runtime` usage in downstream modules | libraries for UGGrid or `libdunecommon` |
| `Documentation` | Documentation files | man pages, Doxygen |
In CMake, objects not assigned to any component are set to `Unspecified` (current behaviour). With the proposed setting, nothing should be in this bucket.
### Usage
Assuming components are set, installation of files in a given component is as simple as:
```bash
# User doesn't want to develop, just to link a final application
cmake --install . --component Library
# User wants to develop a downstream module
cmake --install . --component Library --component Development
# User wants to use final binaries
cmake --install . --component Library --component Runtime
# User wants to use everything (current behaviour)
cmake --install .
```
### Is this backwards compatible?
**Yes!** If no component is given on the command line (current usage), all components are installed. This means that the change is backwards compatible with any installation script out there.
### What next?
Discussion... If there seems to be agreement/interest, I can push a MR.CMake Modernizationhttps://gitlab.dune-project.org/core/dune-common/-/issues/354Compiler warning pollution from dependencies2023-11-06T12:57:49ZAlexander MüllerCompiler warning pollution from dependenciesI tried for my dune module to really go hard on fix warnings from clang.
The output is polluted by giving warnings on all other included modules/header from e.g. `dune-functions,yasp_grid,dune-geometry,...`.
Now I read about the fact th...I tried for my dune module to really go hard on fix warnings from clang.
The output is polluted by giving warnings on all other included modules/header from e.g. `dune-functions,yasp_grid,dune-geometry,...`.
Now I read about the fact that the compilers may not produce warning for system headers.
https://cmake.org/cmake/help/latest/command/target_include_directories.html
>If SYSTEM is specified, the compiler will be told the directories are meant as system include directories on some platforms. This may have effects such as suppressing warnings or skipping the contained headers in dependency calculations (see compiler documentation). Additionally, system include directories are searched after normal include directories regardless of the order specified.
Therefore, in https://gitlab.dune-project.org/core/dune-common/-/blob/1437ba8a2840120ef48d563f798134cbb66db70c/cmake/modules/DuneEnableAllPackages.cmake#L301,
there could be an option to include these as SYSTEM header or even as default?
I'm not sure if this will break.
Maybe there is another mechanism less intrusive I'm unaware of.
Nevertheless, this is a show stopper for me to enable warning, since currently the warning pollution of dependencies is too big.
Any thoughts on this @simon.praetorius @santiago.ospina ? Thanks!https://gitlab.dune-project.org/core/dune-common/-/issues/253What is a matrix?2023-10-17T15:32:05ZSimon PraetoriusWhat is a matrix?## What is a matrix?
While we have CRTP base class `DenseMatrix` specifying the interface for matrices, this base is incomplete or wrong documented and even inconsistent. The specification is also not applied to all matrix-like classes n...## What is a matrix?
While we have CRTP base class `DenseMatrix` specifying the interface for matrices, this base is incomplete or wrong documented and even inconsistent. The specification is also not applied to all matrix-like classes neither in dune-common nor in dune-istl. So, I wanted to open a specific discussion on what is a matrix?
### 1. Matrix is a 2d container of values
On Wikipedia, a matrix is defined as "a rectangular array of numbers (or other mathematical objects)". Thus we have the following requirements:
- (a) A matrix stores elements/entries of a specific type. Seeing matrices as collections, this would mean we have a single element-type. For collections this type is typically denoted by `value_type`.
- "The size of a matrix is defined by the number of rows and columns that it contains". Actually, this definition specifies three size informations:
1. (b) There is a type representing the sizes, typically denoted by `size_type`
2. (c) There is number of rows and number of columns. In Dune this is (for historical reasons) denoted by `N()` and `M()` respectively.
3. (d) There us the *total size* of a matrix. Since a matrix shape is denoted by `n x m`, I would suggest to define the size as the product `n*m`. This would allow to see single-row and single-column matrices as row-vector and column-vector, correspondingly. Compare, e.g., `numpy.matrix.size`, `mtl::size(matrix)`, or `Eigen::Matrix::size`.
- (e) The elements of a matrix should be accessible by indices. Since the row and column indices are in the range `[0,rows)` and `[0,cols)`, its index type should also be `size_type`.
- (f) Following the C-array syntax, elements are accessed by `matrix[i][j]`.
### 2. Matrix is an element of a vector space
Matrices over a field build an algebraic set with vector-space properties:
- matrices for an additive abelian group (commutative group)
- matrices are scalable by scalars of the field type
- Multiplication with scalar is distributive with respect to addition.
These mathematical concepts result in the following requirements for the matrix type `M`:
- There is a field type associate to the matrix, i.e. `Dune::FieldType<M>::field_type` exists.
- There are arithmetic operations for matrices `A` and `B`:
* (g) `A + B`, `A - B`
* (h) `A += B`, `A -= B`
* (i) `-A`, `+A` (optional)
- Multiplication with a scalar `s` of field type:
* (j) `A * s`, `s * A`
* (k) `A *= s`
* (l) `A / s`, `A /= s`
- (m) Additionally, we typically require compound operations: `A.axpy(s, B)`
#### 2.1. Normed space and inner-product space
By defining norms over matrices and inner products, the vector spcae could be turned into a normed
vector space or an inner-product space
- Typical norm for matrices include:
* (n) `frobenius_norm()`
* (o) `forbenius_norm2()` (squared frobenius norm)
* (p) `infinity_norm()`
* (q) `infinity_norm_real()` (simplified infinity norm)
- Inner-products:
* (r) frobenius inner-product `A:B = tr(A^H A)` (not implemented by any matrix)
### 3. Matrix is a linear map
A linear transformation or linear map between vector spaces can be represented by a matrix. The operations of a linear map are typically of the form of matrix-vector products:
- (s) `A.mv(x, y)`: `y = A*x`
- (t) `A.umv(x, y)`: `y+= A*x`
- (u) `A.mmv(x, y)`: `y-= A*x`
- (v) `A.usmv(alpha, x, y)`: `y+= alpha * A*x`
If the matrix additionally provide transposed operations:
- (w) `A.mtv(x, y)`: `y = A^T * x`
- (x) `A.umtv(x, y)`: `y+= A^T * x`
- (y) `A.mmtv(x, y)`: `y-= A^T * x`
- (z) `A.usmtv(alpha, x, y)`: `y+= alpha * A^T * x`
If the matrix additionally provide hermitian operations:
- (A) `A.mhv(x, y)`: `y = A^H * x` (currently not implemented by any matrix)
- (B) `A.umhv(x, y)`: `y+= A^H * x`
- (C) `A.mmhv(x, y)`: `y-= A^H * x`
- (D) `A.usmhv(alpha, x, y)`: `y+= alpha * A^H * x`
### 4. Other operations with matrices
- (E) Matrix multiplication: `A in R^nxm` and `B in R^mxk` then `A * B in R^nxk`
* Matrices must be compatible
* square matrices might form a ring under matrix addition and matrix multiplication
this would allow `A *= B`
- (F) The transposed of a matrix: `A in R^nxm` -> `A^T in R^mxn`.
* Either an operation `A.transpose()` or a free function `transpose(A)`?
* The transposed matrix is itself a matrix
- (G) Trace of a matrix `tr(A) = sum_i A_ii`
* Assumption: matrix is square
* Either an operation `A.trace()` or a free function `trace(A)`?
* Recursive definiton: `tr(A) = sum_i tr(A_ii)`
- (H) Determinant of a matrix
* Assumption: matrix is square
* Easy to define for small matrices
* Generalization using, e.g., Laplace expansion
* Special operation: gramian determinant
- (I) Inversion of a matrix or solution of a linear system
* Assumption: matrix is square
* easy to implement for small matrices
* LU decomposition (or other forms of factorization)
### 5. Matrices can be traversed
Visiting all the (stored) elements of a matrix is a useful operation to generalize some algorithms
- Whats is a proper call for choosing the way/order of traversal?
- It exists `begin()` and `end()` functions.
* Are not documented properly but typically mean traversal of rows.
- Additionally reverse traversal is given by `beforeEnd()` and `beforeBegin()`
* Why not using `rbegin()` and `rend()` as for standard containers?
- Missing functions in the interface:
* Traversal over rows first: e.g. `begin_rows()`, `end_rows()`
* Traversal over columns first: e.g. `begin_cols()`, `end_cols()`
### 6. Special matrix types with more properties
- Sparse matrices:
* Check whether elements exist in the store is required: `A.exists(i,j)`
* Number of stored elements (nonzeros): `A.nonzeroes()`
* Iterators only iterate over stored elements/nonzeros.
* Often direct element-access not implemented (neither const nor mutable)
- Banded/Diagonal matrices:
* Special type of sparse matrix
* Additional access to diagonal(s): `A.diagonal()` (current interface does not allow return off-diagonals)
* Direct access to diagonal entries: `A.diagonal(i)`
- Triangular matrices
- Tridiagonal matrices
## Examples in dune-common
### `DenseMatrix`
A `DenseMatrix` is the CRTB base class for some matrix implementation, but it lacks some of the properties from above. Also
its documentation is partially wrong.
- (c) There are multiple functions returning number of rows/columns: `A.N()` and `A.rows()`, `A.M()` and `A.cols()`, where
`rows() -> mat_rows()` and `cols() -> mat_cols()` call the implementation functions.
- (d) is defined differently. For some reason `A.size()` returns the number of rows.
- (g) arithmetic operations not completely available for matrices, `+A` not implemented (not required)
- (j) and (l) not implemented, except for `A /= s`
- (r) There is no inner product for matrices implemented
- (A) Missing lin-map operation `A.mhv(x,y)`
- (E) binary matrix-matrix product not implemented. For square matrices the mult-assign is implemented from left and
right, i.e., `A.leftmultiply(B)` and `A.rightmultiply(B)`. The general matrix-matrix multiplication is commented out.
- (G) Trace is not implemented.
### `FieldMatrix`
Is an implementation of `DenseMatrix` thus inherits its properties. Some additional comments
- (c) Here `rows` and `cols` are enum constants hiding the interface functions `rows()` and `cols()`
- (g) FieldMatrix implements `A + B` and `A - B`
- (j) and (l) are completely implemented
- (E) Multiplication is implemented as `A * B` and `leftmultiply[any]()` and `rightmultiply[any]()`. Additional operations
are implemented in the namespace `Dune::FMatrixHelp`
### `DynamicMatrix`
Is an implementation of `DenseMatrix` thus inherits its properties.
### `DiagonalMatrix`
- (d) is defined differently. For some reason `A.size()` returns the number of rows.
- (f) The element access is only implemented for diagonal entries, i.e. `matrix[i][j]` throws if `i != j`
- (g), (j) Operations `A + B`, `A - B`, and `-A` not implemented.
- (i) compound assignment operators implemented for scalars: `A += s` and `A -= s` meaning elementwise addition and
subtraction of that scalar.
- (m) Operation `axpy` not implemented.
- (r) There is no inner product for matrices implemented
- (A) Missing lin-map operation `A.mhv(x,y)`
- (E) No matrix-matrix products implemented
- (G) Trace is not implemented.
### `TransposedMatrixWrapper`
The wrapper is created by `transpose(matrix)`, but does not produce a matrix in any sense as described above. It
has just one single operation: `A * B^T`. Note, this wrapper only works for transposing a `FieldMatrix`.
## Test of the interface
There is a test in dune-common: `checkmatrixinterface.hh` but it does not test the complete matrix interface, but just the functions
that were already implemented. Thus, this test is probably written after the implementation of the `DenseMatrix` class and
not before.
An alternative test, that is implemented in terms of separating the various aspects of a matrix, is implemented in !953
in terms of c++20 concepts. These tests can run in the ci jobs for toolchains supporting this new c++ standard and can
report interface failures.
## Major problems with the current interface
The existing matrix implementations in dune-common (but also in dune-istl) only partially fulfil the interface definition
from above and are inconsistent. Major problems are:
- Element access is not a defined property of a matrix. It is only guaranteed to succeed in some matrix implementations.
This is mainly for performance reasons. This argument is a bit weak, because there are other ways to ensure performant
data access, e.g. using iterators or specialized access functions.
- Traversal of matrices is not properly specified and cannot easily be extended to column-major traversal. `begin()` and
`end()` have no knowledge about its traversal direction. There should be a better name for row-wise traversal and
column-wise traversal. Proposal: `begin_rows()` and `begin_cols()`. Then, the user has to explicitly use a range-generator
for the iteration, e.g., `rows(matrix).begin()` and `columns(matrix).begin()`
- The iterators itself are not properly specified. Is the dereferenced iterator again a range? What does it traverse?
- Some matrices implement a single index access `matrix[r]` as a row-view onto the matrix, i.e., this operation returns a
vector-like container representing the `r`th row of the matrix. This is not clearly specified. Is this a required property
of a matrix? What are the costs of this operation? Is the row a propery vector or just a proxy providing minimal access
to the row entries?
- Especially in the implementation of geometries, one needs matrix-matrix multiplication. While this is implemented for
`FieldMatrix`, all the other matrix types lack this operation. But, e.g., `YaspGrid` returns for its jacobian a
`DiagonalMatrix`. Other grid implementations maybe something else. We need `A * B`, but also `A^T * B` and `A * B^T`
for various combinations of matrices. This is partially implemented in some "Helper" namespaces in dune-geometry or
for `FieldMatrix` in `FMatrixHelper` namespace. Multiplication seems not properly designed!
- Sparse matrices are a special type of matrices. The interface should mirror this mathematical property. Actually, sparse
matrices are just matrices with a different storage type for its entries. Thus, sparse matrices should only extend the
matrix interface with methods providing fast access to the stored elements. Either by iterators or something else. Currently,
sparse matrices only implement a partial matrix interface and some methods even have different behaviour, e.g. `matrix[i][j]`
is not guaranteed to succeed, not even for const-access, although the matrix value is clearly specified.
- Views over matrices, like transposed-view or hermitian-view are not properly implemented. They do not provide a matrix
interface although this could easily be provided. One could discuss whether views are read-only or provide mutable
access to the matrix elements as well.
- Currently, we can only test that a matrix supports all the different aspects as described above. Not the individual aspect. But, often an algorithm just needs some properties, e.g., in iterative krylov methods, one needs the linear-map properties for a matrix, for lu-decomposition maybe fast element access. In geometry parametrizations we need matrix-matrix multiplication and trace/Gramian determinant computation. It is required to have individual concept/interface checks.
- Single-row and Single-column matrices are not vectors. This often leads to problems, e.g., there is no `two_norm()` implemented and also `dot()` and `axpy` with another vector does not work, even if the shape would allow this. This problem is that vectors are not matrices and also not compatible in its interface to matrices.https://gitlab.dune-project.org/core/dune-common/-/issues/345Document how to make interdependent targets on different namespaces2023-10-05T11:35:07ZSantiago Ospina De Los Ríossospinar@gmail.comDocument how to make interdependent targets on different namespacesThe following discussion from !1247 should be addressed:
- [ ] @simon.praetorius started a [discussion](https://gitlab.dune-project.org/core/dune-common/-/merge_requests/1247#note_129022): (+13 comments)
> Unfortunately, this does...The following discussion from !1247 should be addressed:
- [ ] @simon.praetorius started a [discussion](https://gitlab.dune-project.org/core/dune-common/-/merge_requests/1247#note_129022): (+13 comments)
> Unfortunately, this does not work properly. If one target depends on the other target (e.g. via `target_link_libraries`), the dependency must be included before. Otherwise I get the fatal error "The following imported targets are referenced, but are missing..." automatically generated in the target file. Not sure how to fix this properly. I have to create the libraries in the right order, I think, and then it works. But this is not the way to want to resolve dependencies.https://gitlab.dune-project.org/core/dune-common/-/issues/329Build requirements are not properly documented2023-09-23T10:46:42ZCarsten Gräsergraeser@math.fau.deBuild requirements are not properly documentedCurrently the build instructions in `README.md` and `INSTALL` are missing a lot of very important information that should be added:
* An internet connection is needed because `pip` is used to download python packages.
* When no connecti...Currently the build instructions in `README.md` and `INSTALL` are missing a lot of very important information that should be added:
* An internet connection is needed because `pip` is used to download python packages.
* When no connection is available, one gets several warnings and error messages.
However, the build seems to terminate successfully. Furthermore CMake says that the optional package `Python3`has been found.
Either this is not a problem and the build is successful, then we should not scare users with these error messages. Or this
means that the python-bindings cannot be used than this should be properly reported to the user.
* If the internet connection is available, it is used to download executable code from third party locations that is executed later.
* Since the required python modules are not documented it's also not possible that the user provides them on his own
to avoid this dangerous behavior. Notice that even if the python packages mentioned in the error messages are installed
globally, the attempt to download them and the mentioned error messages persists.
BTW:
One can argue that downloading and executing code from third party locations is a no-go unless the user explicitly opts in.
This ticket is not about changing this behavior now. But we should at least be honest and clearly document this and tell the
users how it can be disabled.Markus BlattMarkus Blatthttps://gitlab.dune-project.org/core/dune-common/-/issues/338[Python] Create Python bindings when Python package has different name to dun...2023-09-22T14:01:44ZAlexander Müller[Python] Create Python bindings when Python package has different name to dune moduleI'm trying and struggling currently to create working python bindings for python bindings, where the pypi package should have a different name to the dune module and to the actual python package.
I'm using all 2.9 branches
I.e. my dune...I'm trying and struggling currently to create working python bindings for python bindings, where the pypi package should have a different name to the dune module and to the actual python package.
I'm using all 2.9 branches
I.e. my dune-module is called "ikarus" which is not available at pypi. Thus, as a workaround I want to name the pypi package `pyikarus`.
Thus I want the behaviour to be `pip install pyikarus` and in the python code `import ikarus` and my cpp folder structure should also be with `ikarus`.
I investigated, and my current setup is:
In the top-level `setup.py` I have
```python
...
metadata["name"] = "pyikarus"
setup(**metadata)
```
In `python/ikarus/setup.py.in` I have
```python
REQUIRED_PACKAGES = '${RequiredPythonModules}'.replace(';',' ').split(' ')
setup(
name="pyikarus",
description="${ProjectDescription}",
version="${ProjectVersionString}",
author="${ProjectAuthor}",
author_email="${ProjectMaintainerEmail}",
packages=find_packages(exclude=["docs/*"]),
zip_safe=0,
package_data={"": ["*.so"], "pyikarus": ["data/*.cmake"]},
install_requires=REQUIRED_PACKAGES,
include_package_data=True,
)
```
and in `python/ikarus/CMakeLists.txt`
```cmake
add_subdirectory(ikarus)
set(CXX_MAX_STANDARD 20)
set(PYPI_PACKAGE_NAME "pyikarus")
dune_python_configure_bindings(
PATH
"."
PACKAGENAME
${PYPI_PACKAGE_NAME}
CMAKE_METADATA_FLAGS
DUNE_OPTS_FILE
CXX_MAX_STANDARD
)
if(POLICY CMP0087)
cmake_policy(SET CMP0087 NEW)
endif()
```
There are several points where I have to state `ikarus` or `pyikarus`. Brute forcing some variants of these options always returns errors.
Thus, if I install my uploaded package from pypi and try to compile my code with `dune-py`, I get two different error messages.
If I pass to `dune_python_configure_bindings` above the name `pyikarus`, I get errors like
<details>
<summary>pyikarus error message</summary>
```python
---------------------------------------------------------------------------
CompileError Traceback (most recent call last)
Cell In[35], line 1
----> 1 svk = iks.StVenantKirchhoff(emodul=1000,nu=0.3)
3 svkPS = svk.asPlainStress()
File /dune/dune-common/build-cmake/dune-env/lib/python3.10/site-packages/ikarus/materials.py:33, in materialConstructorDecorator..wrapper(emodul, nu)
31 includes += ["ikarus/python/finiteElements/materials/material.hh"]
32 moduleName = func.__name__+"_" + hashIt(element_type)
---> 33 module = generator.load(
34 includes=includes,
35 typeName=element_type,
36 moduleName=moduleName
37 )
39 # dispatcher = { 'NeoHooke' : NeoHooke, 'StVenantKirchhoff' : StVenantKirchhoff}
42 return eval("module."+func.__name__+"(emodul,nu)")
File /dune/dune-common/build-cmake/python/dune/generator/generator.py:172, in SimpleGenerator.load(self, includes, typeName, moduleName, extraCMake, defines, preamble, postscript, options, bufferProtocol, dynamicAttr, baseClasses, holder, *args)
168 for nr, (tn, a, o, b, d, bc, h) in enumerate( zip(typeName, args, options, bufferProtocol, dynamicAttr, baseClasses, holder) ):
169 source += self.main(nr, includes, tn, *a, options=o,
170 bufferProtocol=b, dynamicAttr=d,
171 baseClasses=bc, holder=h)
--> 172 return self.post(moduleName, source, postscript, extraCMake)
File /dune/dune-common/build-cmake/python/dune/generator/generator.py:124, in SimpleGenerator.post(self, moduleName, source, postscript, extraCMake)
122 # make sure to reload the builder here in case it got updated
123 from . import builder
--> 124 module = builder.load(moduleName, source, self.typeName[0], extraCMake)
126 return module
File /dune/dune-common/build-cmake/python/dune/generator/cmakebuilder.py:382, in Builder.load(self, moduleName, source, pythonName, extraCMake)
380 module = sys.modules.get("dune.generated." + moduleName)
381 if module is None:
--> 382 self._buildModule( moduleName, source, pythonName, extraCMake )
384 ## TODO remove barrier here
385 comm.barrier()
File /dune/dune-common/build-cmake/python/dune/generator/cmakebuilder.py:732, in MakefileBuilder._buildModule(self, moduleName, source, pythonName, extraCMake)
729 # check return code
730 if exit_code > 0:
731 # retrieve stderr output
--> 732 raise CompileError(buffer_to_str(stderr))
CompileError: /dune/dune-common/build-cmake/dune-env/.cache/dune-py/python/dune/generated/StVenantKirchhoff_de7f96cf2f383220c197a4a670079062.cc:10:10: fatal error: ikarus/finiteElements/mechanics/materials.hh: No such file or directory
10 | #include
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
```
</details>
Thus, the underlying cmake script does maybe not include the correct subdirectories to make the headers findable.
If I pass to `dune_python_configure_bindings` above the name `ikarus` I get errors like
<details>
<summary>pyikarus error message</summary>
```python
---------------------------------------------------------------------------
CompileError Traceback (most recent call last)
Cell In[4], line 1
----> 1 svk = iks.StVenantKirchhoff(emodul=1000,nu=0.3)
3 svkPS = svk.asPlainStress()
File /dune/dune-common/build-cmake/dune-env/lib/python3.10/site-packages/ikarus/materials.py:33, in materialConstructorDecorator..wrapper(emodul, nu)
31 includes += ["ikarus/python/finiteElements/materials/material.hh"]
32 moduleName = func.__name__+"_" + hashIt(element_type)
---> 33 module = generator.load(
34 includes=includes,
35 typeName=element_type,
36 moduleName=moduleName
37 )
39 # dispatcher = { 'NeoHooke' : NeoHooke, 'StVenantKirchhoff' : StVenantKirchhoff}
42 return eval("module."+func.__name__+"(emodul,nu)")
File /dune/dune-common/build-cmake/python/dune/generator/generator.py:172, in SimpleGenerator.load(self, includes, typeName, moduleName, extraCMake, defines, preamble, postscript, options, bufferProtocol, dynamicAttr, baseClasses, holder, *args)
168 for nr, (tn, a, o, b, d, bc, h) in enumerate( zip(typeName, args, options, bufferProtocol, dynamicAttr, baseClasses, holder) ):
169 source += self.main(nr, includes, tn, *a, options=o,
170 bufferProtocol=b, dynamicAttr=d,
171 baseClasses=bc, holder=h)
--> 172 return self.post(moduleName, source, postscript, extraCMake)
File /dune/dune-common/build-cmake/python/dune/generator/generator.py:124, in SimpleGenerator.post(self, moduleName, source, postscript, extraCMake)
122 # make sure to reload the builder here in case it got updated
123 from . import builder
--> 124 module = builder.load(moduleName, source, self.typeName[0], extraCMake)
126 return module
File /dune/dune-common/build-cmake/python/dune/generator/cmakebuilder.py:374, in Builder.load(self, moduleName, source, pythonName, extraCMake)
370 def load(self, moduleName, source, pythonName, extraCMake=None):
371 # check if we need to initialize dune-py either because
372 # this is the first call to load or because an external module with metadata has been registered
373 if not self.initialized or not self.externalPythonModules == getExternalPythonModules():
--> 374 self.initialize()
376 # check whether module is already compiled and build it if necessary
377 # (only try to build module on rank 0!)
378 # TODO replace if rank with something better and remove barrier further down
379 if comm.rank == 0:
File /dune/dune-common/build-cmake/python/dune/generator/cmakebuilder.py:616, in MakefileBuilder.initialize(self)
615 def initialize(self):
--> 616 super().initialize()
617 # check that the compile script is available
618 script = os.path.join(self.generated_dir,"buildScript.sh")
File /dune/dune-common/build-cmake/python/dune/generator/cmakebuilder.py:213, in Builder.initialize(self)
211 self.cacheExternalModules()
212 # create dune-py module
--> 213 self.build_dunepy_from_template(self.dune_py_dir)
214 # create tag file so that dune-py is not rebuilt on the next build
215 open(tagfile, 'a').close()
File /dune/dune-common/build-cmake/python/dune/generator/cmakebuilder.py:472, in MakefileBuilder.build_dunepy_from_template(dunepy_dir, force)
468 if not useNinja:
469 # call base class dunepy_from_template (re-initialize)
470 force = Builder.generate_dunepy_from_template(dunepy_dir, force=True)
--> 472 Builder.callCMake(["cmake"] + defaultCMakeFlags() + ["."],
473 cwd=dunepy_dir,
474 infoTxt="Configuring dune-py with CMake (make)",
475 active=True, # print details anyway
476 )
478 # obtain bash and make command generated by cmake
479 # when ninja is used makeCmd is the ninja command
480 try:
File /dune/dune-common/build-cmake/python/dune/generator/cmakebuilder.py:263, in Builder.callCMake(cmake_args, cwd, infoTxt, verbose, active, logLevel, env)
260 if cmake.returncode > 0:
261 # retrieve stderr output
262 print(stderr,stdout)
--> 263 raise CompileError(buffer_to_str(stderr))
265 return stdout, stderr
CompileError: ---- LOCK
---- ACQUIRED
-- Generating CXX compiler script for CXXFLAGS overloading on command line
---- RELEASE
---- DONE
CMake Error at /dune/dune-common/build-cmake/dune-env/lib/cmake/ikarus/ikarus-targets.cmake:61 (set_target_properties):
The link interface of target "ikarus" contains:
autodiff::autodiff
but the target was not found. Possible reasons include:
* There is a typo in the target name.
* A find_package call is missing for an IMPORTED target.
* An ALIAS target is missing.
Call Stack (most recent call first):
/dune/dune-common/build-cmake/dune-env/lib/cmake/ikarus/ikarus-config.cmake:55 (include)
/dune/dune-common/cmake/modules/DuneModuleDependencies.cmake:170 (find_package)
/dune/dune-common/cmake/modules/DuneModuleDependencies.cmake:289 (find_dune_package)
/dune/dune-common/cmake/modules/DuneModuleDependencies.cmake:336 (dune_process_dependency_leafs)
/dune/dune-common/cmake/modules/DuneModuleDependencies.cmake:46 (dune_create_dependency_leafs)
/dune/dune-common/cmake/modules/DuneProject.cmake:90 (dune_create_dependency_tree)
CMakeLists.txt:106 (dune_project)
```
</details>
which indicates by the linking error, that in some cmake file the linking is not done correctly, thus indicating the first version with `pyikarus` is better? But anyway the headers are not findable in this first case. Changes in the files in `python/ikarus/` does not change the error messages.
Can someone please point me to the correct changes. I'm willing to create then a proper MR to enable this functionality, if this is of interest. Otherwise, if this should already work can someone correct my mistakes?
The current folder structure can be found at https://github.com/ikarus-project/ikarus/tree/feature/pythonBindingsV3.
Thanks, and any help is very much appreciated.Alexander MüllerAlexander Müllerhttps://gitlab.dune-project.org/core/dune-common/-/issues/198Status of TBB2023-09-20T07:17:05ZNils-Arne DreierStatus of TBBI'm planing to use TBB for thread parallelism in a ISTL based project. I'm willing to prepare a MR in dune-istl when I introduced all the TBB calls. But I'm wondering what is the actual status of TBB in dune?
The build infrastructure is ...I'm planing to use TBB for thread parallelism in a ISTL based project. I'm willing to prepare a MR in dune-istl when I introduced all the TBB calls. But I'm wondering what is the actual status of TBB in dune?
The build infrastructure is there but it is not used in the core modules.
On the 2017 developer meeting was a discussion about that: https://dune-project.org/community/meetings/2017-03-devmeeting/threading-infrastructure/
That is what was agreed on:
> We could introduce an abstraction layer. But that only makes sense if we have at least a rough idea regarding the possible variability of the interface. We do not know about any other possible interface. And wrapping will in some cases be quite expensive (support wise)
and
>We standardise on TBB without abstraction/wrappers. TBB remains optional: if not available there will be not automatic threading in the core modules. Infrastructure that only makes sense in connection with TBB may be missing then. - We do not allow OpenMP in core modules. Users may use OpenMP in their own modules. - We generally disallow threads not managed by the thread pool. In special cases you may start threads outside (e.g. for communicating in the background), but you need to ensure that there are no adverse effects, e.g. performace loss due to oversubscription. - When there is equivalent functionality in TBB and the C++ standard, prefer the standard functionality.
Does that mean, that whenever I want to use TBB in the core modules if have to provide a fallback like
```c++
#if HAVE_TBB
tbb::parallel_for(..., [](...){
...
}):
#else
for(...){
...
}
#endif
```
I think that would lead to very ugly code. Should we provide at least convenient functions for the most used cases like `parallel_for`?https://gitlab.dune-project.org/core/dune-common/-/issues/343Follow-up from "[cleanup][OverloadCompilerFlags] Move compiler flag overloadi...2023-09-18T21:16:45ZAndreas DednerFollow-up from "[cleanup][OverloadCompilerFlags] Move compiler flag overloading to user script."The following discussion from !1251 should be addressed:
- [ ] @santiago.ospina started a [discussion](https://gitlab.dune-project.org/core/dune-common/-/merge_requests/1251#note_128976): (+26 comments)
> @robert.kloefkorn There i...The following discussion from !1251 should be addressed:
- [ ] @santiago.ospina started a [discussion](https://gitlab.dune-project.org/core/dune-common/-/merge_requests/1251#note_128976): (+26 comments)
> @robert.kloefkorn There is some errors in the CI for [dune-istl](https://gitlab.dune-project.org/core/dune-istl/-/jobs/491718) and [dune-ugrid](https://gitlab.dune-project.org/staging/dune-uggrid/-/jobs/490354) wrt the configuration of dune-py. The only change in between the failing pipelines seems to be this MR. The error seems to be that the generated cmake has/non-existing wrong parameters.https://gitlab.dune-project.org/core/dune-common/-/issues/266Merge-Conflicts in CHANGELOG.md2023-06-12T07:28:48ZSimon PraetoriusMerge-Conflicts in CHANGELOG.md### Merge Conflicts
Whenever someone adds a new feature, fixes a bug, deprecates a function or removes something from the code, a changelog entry should be created. This actually means, nearly each MR adds a line into the `CHANGELOG.md`....### Merge Conflicts
Whenever someone adds a new feature, fixes a bug, deprecates a function or removes something from the code, a changelog entry should be created. This actually means, nearly each MR adds a line into the `CHANGELOG.md`. If there are two MRs at the same time (what is always the case) and one gets merged, the second cannot be merged without rebasing to the latest commit. This is frustrating and increases the development-time.
Can we solve this problem somehow?
There was a similar discussion going on in the group of gitlab developers, see a summary of this discussion process here: https://about.gitlab.com/blog/2018/07/03/solving-gitlabs-changelog-conflict-crisis/
The solution they propose is an automatic changelog tool. But the steps before are: instead of directly filling up a `CHANGELOG.md`, add entries into separate .yml files in a directory, e.g., `CHANGELOG/unreleased/`. These entries are then merged into one markdown file in the release process. There are some subtleties in this process, but those could be solved.
Another idea would be: Add the changelog entry only after the MR is reviewed, directly before there merge. Then there might be less merge conflicts but we need to remember to do this. It has another consequence: The additional commit of an entry in the CHANGELOG file requires another CI pipeline run. This is a wast of computing resources, but also increases the time for a merge. A workaround of this problem could be to add an exception to the `.gitlab-ci.yml` file that when some listed files are changed only, no pipeline will be created.
### Changelog format
Another aspect I have learned from reading about changelog files is to use a proper changelog format for better following the changes, see, e.g., https://keepachangelog.com/en/1.1.0/ or https://docs.gitlab.com/ee/development/changelog.html. The summary is: sort changelog entries in some categories, e.g.,
- added: New feature
- fixed: Bug fix
- changed: Feature change
- deprecated: New deprecation
- removed: Feature removal
- performance: Performance improvement
- other: Other
What do you think about this? Should we improve the handling of changelogs or is this "work without benefits"?https://gitlab.dune-project.org/core/dune-common/-/issues/137[parametertree] allow weak reference / sym-links in ParameterTree2023-04-27T16:54:53ZChristian Engwer[parametertree] allow weak reference / sym-links in ParameterTreeThere are a couple of use-cases for references in the tree. After a first attempt this issue should collect a proper definition of the requirements and structure the discussion (see also !580).
Status quo:
- The `ParameterTree` models ...There are a couple of use-cases for references in the tree. After a first attempt this issue should collect a proper definition of the requirements and structure the discussion (see also !580).
Status quo:
- The `ParameterTree` models a hierarchic structure of sections, similar to a nested dict in python.
- Leafs are values, where one particular kind of values is also an array.
- This data-model differs from the often used json-data-model (!), where dicts and arrays can be nested.
Requirements:
* Aliasing a section to another section
* Aliasing a value to another value
* Alias target may be defined after the alias
* An alias should track the value of the target
* Aliasing should work across parses (i.e. an alias defined on the command line should be able to reference a section in a parameter file read later)
* keep compatible to the current API, of course
Open questions:
1. Do we need cycles in the tree or can we for this thing to model a DAG?
* in case of cycles, we need to keep track of what is a reference and what not.
* in case of a DAG we have to check for cycles during parsing an bail out in case.
1. Do we need to allow modifying values?
* The current `get` method does not allow to modify the internal data, where as the `operator[]` is in my opinion more an internal interface for constructing a tree
* if we don't support modification, we might avoid real references by copying values during parsing
1. Do we want to support references as sub-expressions of a value, e.g.
```
[config]
zslice = 0.3
position = ( 1.0, 10.3, @zslice )
label = "foo".@zslice
```
1. How flexible does the reference need to be?
* There is a proposal for JSON (json-path) using a similar syntax to XPath.
* How to reference relative paths?
* How to reference parents?
* Do we need something like 7th-sibling?
* I tend to to the idea to keep it simple and just use relative and absolute paths.
1. Which syntax to we want to use (in the ini-file parser)
* current suggestion is
* `[foo = bar]` for a section alias foo referencing the section bar
* `foo = @sec.bar` for a key foo mapping to the key bar in section sec
* we might want to unify both and use `[foo = @bar]`ParameterTree overhaulthttps://gitlab.dune-project.org/core/dune-common/-/issues/330[discussion] drop or simplify pip & venv magic during configure2023-04-27T14:33:51ZChristian Engwer[discussion] drop or simplify pip & venv magic during configureAs we have seen in !1235, the automagic handling of python dependencies leads to a couple of strange problems, e.g. packages installed automatically by `cmake` are not always found.
I would like to start a discussion on how we can simpl...As we have seen in !1235, the automagic handling of python dependencies leads to a couple of strange problems, e.g. packages installed automatically by `cmake` are not always found.
I would like to start a discussion on how we can simply this whole cmake-venv-pip integration. My provocative statement is that it leads to more problems than it solves and that a clear error message about missing dependencies would be far more helpful.
Opinions?https://gitlab.dune-project.org/core/dune-common/-/issues/325dune-env and pip: Can it be disabled?2023-04-09T14:13:57ZRobert Kdune-env and pip: Can it be disabled?Is there a simple way to disable pip during dunecontrol runs?
This causes problems when building DUNE on parallel clusters where one has to use the compute node also building and then this compute note does not have internet access. Fo...Is there a simple way to disable pip during dunecontrol runs?
This causes problems when building DUNE on parallel clusters where one has to use the compute node also building and then this compute note does not have internet access. For some reason, CMake refused to believe that pip was indeed installed and would try to download and install the modules again (which would fail because of no internet).https://gitlab.dune-project.org/core/dune-common/-/issues/311python packaging fails with strange errors if dependencies are not available ...2023-04-09T14:10:39ZChristian Engwerpython packaging fails with strange errors if dependencies are not available in the pipy repositoryI tried building locally a python package for dnue-uggrid.
The current build infrastructure tries to install the dependencies. The problem is now that `dune-uggrid` is version 2.9 and requires `dune-common` version 2.9. My initial idea ...I tried building locally a python package for dnue-uggrid.
The current build infrastructure tries to install the dependencies. The problem is now that `dune-uggrid` is version 2.9 and requires `dune-common` version 2.9. My initial idea was to install `dune-common` 2.9 into my local venv using `pip install ./dune-common` which worked perfectly:
```
> pip list
Package Version
------------------ ---------------
certifi 2022.6.15
charset-normalizer 2.0.12
cmake 3.22.5
distro 1.7.0
dune-common 2.9.dev20220622
idna 3.3
Jinja2 3.1.2
MarkupSafe 2.1.1
mpi4py 3.1.3
ninja 1.10.2.3
numpy 1.22.4
packaging 21.3
pip 22.1.1
portalocker 2.4.0
pyparsing 3.0.9
requests 2.28.0
scikit-build 0.15.0
setuptools 59.6.0
urllib3 1.26.9
wheel 0.37.1
```
... but still `pip install ./dune-uggrid` tries to install `dune-common` and finds only the 2.8 wheel:
```
> pip -v install dnue-uggrid/
Using pip 22.1.1 from /home/christi/Uni/Dune/foobar/lib/python3.10/site-packages/pip (python 3.10)
Processing ./uggrid
Running command pip subprocess to install build dependencies
Collecting cmake>=3.13
Using cached cmake-3.22.5-py2.py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (22.5 MB)
Collecting dune-common
Using cached dune_common-2.8.0.post2-cp310-cp310-linux_x86_64.whl
Collecting ninja
Using cached ninja-1.10.2.3-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
Collecting pip
Using cached pip-22.1.2-py3-none-any.whl (2.1 MB)
Collecting requests
Using cached requests-2.28.0-py3-none-any.whl (62 kB)
Collecting scikit-build
Using cached scikit_build-0.15.0-py2.py3-none-any.whl (77 kB)
Collecting setuptools
Using cached setuptools-62.6.0-py3-none-any.whl (1.2 MB)
Collecting wheel
Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
Collecting portalocker
Using cached portalocker-2.4.0-py2.py3-none-any.whl (16 kB)
Collecting mpi4py
Using cached mpi4py-3.1.3-cp310-cp310-linux_x86_64.whl
Collecting numpy
Using cached numpy-1.22.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.8 MB)
Collecting urllib3<1.27,>=1.21.1
Using cached urllib3-1.26.9-py2.py3-none-any.whl (138 kB)
Collecting idna<4,>=2.5
Using cached idna-3.3-py3-none-any.whl (61 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2022.6.15-py3-none-any.whl (160 kB)
Collecting packaging
Using cached packaging-21.3-py3-none-any.whl (40 kB)
Collecting distro
Using cached distro-1.7.0-py3-none-any.whl (20 kB)
Collecting pyparsing!=3.0.5,>=2.0.2
Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB)
Installing collected packages: ninja, cmake, wheel, urllib3, setuptools, pyparsing, portalocker, pip, numpy, mpi4py, idna, distro, charset-normalizer, certifi, requests, packaging, dune-common, scikit-build
```
The build will then later fail with a `cmake` error, as it only find `dune-common` version 2.8 and not version 2.9.
If this intended behaviour? Or how should I properly handle this local install?https://gitlab.dune-project.org/core/dune-common/-/issues/300GenerateTypeName is fragile2023-04-09T14:08:30ZChristian EngwerGenerateTypeName is fragileThe `GenerateTypeName` infrastructure in `typeregistry.hh` requires detailed knowledge about the details of the C++ type. Providing these manually is error-prone, see e.g. the global basis in dune-functions (dune-functions#70).
The glob...The `GenerateTypeName` infrastructure in `typeregistry.hh` requires detailed knowledge about the details of the C++ type. Providing these manually is error-prone, see e.g. the global basis in dune-functions (dune-functions#70).
The global basis defines a `DiscreteGlobalBasisFunction`, but sets a type name to `GlobalBasis::DiscreteFunction`. When trying to generate C++ code the can operate on such a discrete function, we get compiler errors, because there is no typename `GlobalBasis::DiscreteFunction` in the concrete `GlobalBasis`.
Couldn't we make `GenerateTypeName` more robust by using `className` to directly extract the correct type name?
The problem is that `GenerateTypeName` is hardly documented, thus I'm unsure about implications of such a change.https://gitlab.dune-project.org/core/dune-common/-/issues/279Python packages should not be installed during configure and make2023-04-09T14:02:35ZTimo KochPython packages should not be installed during configure and makeThe usual procedure of installing C++ based projects is configure, make, install. With the new Python bindings enabled per default there some IMO __unexpected behaviour__:
1. During `configure` (e.g. `dunecontrol configure`) Python pack...The usual procedure of installing C++ based projects is configure, make, install. With the new Python bindings enabled per default there some IMO __unexpected behaviour__:
1. During `configure` (e.g. `dunecontrol configure`) Python packages (like `jinja2`, `numpy`) are silently installed into the virtual environment (external if a venv activated, internal in dune-common if we are not in a venv). This doesn't even produce output.
2. During `make all` (e.g. `dunecontrol make`) Python packages are installed into the virtual environment. This time it's the Dune packages depending on the Python bindings.
3. During `make install_python` packages get installed globally (system-wide) even if a virtual env is activated. EDIT: This only happens when the internal venv is used.
__Expected behaviour__:
1. `configure` only configures (and doesn't create a venv)
2. `make all` only builds (also building Python bindings (the C++ part) that are enabled per default now)
3. `make install_python` installs Python packages (system-wide may be the default but if a venv is actived it would be nice to install it in there per default)
4. `make install_python_editable` installs Python packages editable instead (would be nice to have)
__Some current issues__
1. It currently doesn't seem possible to just configure and build without installing Python modules. This requires an internet connection (ok it prints some warnings if there is no internet and skips it I think). I might want to just build my code locally without an internet connection and compile some C++ stuff. Why do I need to install Python bindings for that? Or start my own virtual env after running configure. Then the packages are installed in a now useless internal venv. Why not wait?
2. Or in a CI pipeline, I might want to separate Python and C++ testing in different build jobs where the C++ job doesn't need to know anything about the Python job.
3. I might want to install my Python modules system-wide with `make install_python`. However, per default everything always gets installed editable inside the internal venv during `make` already, although this is completely unnecessary in that case.
4. In the case of internal venv the build dependency / flow of information is inverted. Usually, downstream modules depend on upstream modules. Now, the dune-common build directory contains information from all downstream modules.
5. Dune now (at least on Ubuntu) requires to install `python-venv` otherwise the dunecontrol fails.
Building the Python C++ binding modules per default is of course ok. But why is the __installation__ of Python packages not optional anymore (like in Dune 2.8)? Also is there a reason not to delay the installation until installed Python packages are actually needed?
__Suggestion__:
1. Don't create an internal virtual env during configure or build/make. If an external venv is activated, use that as default installation destination, if not, install system-wide per default (like for C++) ((_alternatively, default-install into an internal documented location like now but then `make install_python` should create the venv and install there and not `configure` or `make all`_))
2. Configure only configures (includes configuration of Python packages which depend on CMake information), and one can pass an installation path for the Python packages (if you don't want to install system-wide or in an external activated venv)
3. `make all` only builds libraries (no C++ tests, and no Python installation)
4. `make install_python_editable` installs Python packages into configured destination in editable mode using `pip`
5. `make install_python` installs Python packages into configured destination in normal mode using `pip`
6. `make install`? I have no idea what this currently does in combination with Python packages.
I think this would make things both simpler to implement, more transparent, consistent and expected.
__Question__:
Is the following workflow already possible (by setting some variables) but I just don't know it?:
1. disable creation of internal venv and provide path to external venv via CMake (that is not activated yet)
2. run `dunecontrol all` which _only_ configures and builds _without_ installing like in 2.8 with Python bindings enabled
3. run some install command that installs Python modules and depedencies in provided path (possibly in editable mode)
I think this is at least one reasonable workflow that should be supported.
__User perspective: suggestion vs status quo__:
I think in the minimal case the following would be possible: (Assuming I cloned all modules into a directory)
* Currently (master), to run a python script with dune is as simple as:
- `./dune-common/bin/dunecontrol --opts=my.opts all`
- `source ./dune-common/build-cmake/dune-env/bin/activate`
- `python my_python_test.py`
* With what is suggested here, I believe, the same could be achieved with:
- `./dune-common/bin/dunecontrol --opts=my.opts all`
- `./dune-common/bin/dunecontrol make install_python_editable`
- `python my_python_test.py`
__Related__:
* Even with DUNE_ENABLE_PYTHONBINDINGS=0 a virtual env is installed. Would be nice to have one variable to turn off the Python stuff. (also see #293)https://gitlab.dune-project.org/core/dune-common/-/issues/273[Python] cmake not needed for install but for running scripts2023-04-09T13:58:44ZAndreas Dedner[Python] cmake not needed for install but for running scripts`cmake` is a python install requirement in the `pyproject.toml` so the installation of the Python packages works fine. But cmake is not part of the 'python-require' list in `dune.module` so cmake might not be available when building `dun...`cmake` is a python install requirement in the `pyproject.toml` so the installation of the Python packages works fine. But cmake is not part of the 'python-require' list in `dune.module` so cmake might not be available when building `dune-py`.
After adding `cmake` to the requirements it would be installed into `dune-env/bin` and then should be available for building `dune-env`. We could also add `ninja` and always use that for `dune-py`.https://gitlab.dune-project.org/core/dune-common/-/issues/316Some methods in Communication<T> are unnecessarily mutable2023-03-14T14:12:07ZSantiago Ospina De Los Ríossospinar@gmail.comSome methods in Communication<T> are unnecessarily mutableSome methods like `send` and `recv` in the dummy `Communication<T>` are mutable while their counterpart in `Communication<MPI_Comm>` are const. Although copying the object is not a problem, I don't see the reason why these have to be mut...Some methods like `send` and `recv` in the dummy `Communication<T>` are mutable while their counterpart in `Communication<MPI_Comm>` are const. Although copying the object is not a problem, I don't see the reason why these have to be mutable.https://gitlab.dune-project.org/core/dune-common/-/issues/262test/eigenvaluetest.cc fails on ARM64 machines2023-03-14T12:49:13ZPatrick Jaaptest/eigenvaluetest.cc fails on ARM64 machinesThe Debian build for ARM64 fails due to a too strict check for `long double` in the line
```
if((Av - eigenValues[j]*eigenVectors[j]).two_norm() > dim*std::sqrt(std::numeric_limits<field_type>::epsilon()))
```
where we compare
`6.56817e-...The Debian build for ARM64 fails due to a too strict check for `long double` in the line
```
if((Av - eigenValues[j]*eigenVectors[j]).two_norm() > dim*std::sqrt(std::numeric_limits<field_type>::epsilon()))
```
where we compare
`6.56817e-16 > 5.55112e-17`
I see no trivial way how to improve this check. Therefore I decided to open an issue here.
Full Debian build log here (does not contain useful information):
https://buildd.debian.org/status/fetch.php?pkg=dune-common&arch=arm64&ver=2.8.0%7Erc1-1&stamp=1629831577&raw=0https://gitlab.dune-project.org/core/dune-common/-/issues/174Communication Patterns.2023-03-13T21:11:25ZRobert KCommunication Patterns.Basis for discussion at the dev meeting
**Motivation**
- keep data distribution independent from algorithms
- different sources for data distribution:
- Grid + Discretization(MCMGMapper)
- ParMetis
- Global IndexSets
- some dist...Basis for discussion at the dev meeting
**Motivation**
- keep data distribution independent from algorithms
- different sources for data distribution:
- Grid + Discretization(MCMGMapper)
- ParMetis
- Global IndexSets
- some distributed fancy matrix format
- ...
**Owner - Overlap - Copy semantic**
- Owner: disjunct decomposition of DoFs
- Owner ∪ Overlap: DoFs of the local Problem (Domain decomposition)
- Copy: Auxiliary DoFs
**Requirements**
- Scalar product: sum over owner DoFs
- Operator:
- Overlapping: copies owner DoFs to corresponding remote DoFs
- Nonoverlapping: sums all Dofs up
- ... (?)
- Preconditioner:
- Domain Decomposition (Schwarz like): project to local problem - set all Copy DoFs to 0.
- AMG: ?
**Stored Information**
- Every local index has a tag $`t\in T=(\text{Owner, Overlap, Copy})`$ and a list of tuples $`R\subset \mathbb{N}\times T`$ containing remote ranks that store this DoF as well and their tags.
- If $`R`$ is empty, $`t`$ must be Owner -> Don't store anything
**Interface**
- ScalarProduct:
- Currently in ISTL: mask with 1 on Owners and 0 else
- Better: List of non-Owner indices: iterate in between
- Data Consistency: (Copy|Sum) FROM localTags TO remoteTags
**Problems**
- Hierarchical indices?!
- Do we want to allow other tags that owner,overlap,copy
<!-- Old stuff
TODOs:
- [X] Communication pattern holding InterfaceInformation (or std::vector< int >) for send and receive ranks
- [X] simple test in dune-common setting up a communication pattern
- [ ] convertion from `RemoteIndexSet` to `CommunicationPattern`
- [X] PatternCommunicator obtaining pattern and using MPIPack buffer
- [X] test for PatternCommunicator
- [ ] Communication pattern based on MCMGMapper
- [ ] simple test in dune-grid for MCMGMapper building comm pattern
- [ ] in dune-istl adjustment to PatternCommunicator
- [ ] Projection operator for copy dofs
- [ ] test in dune-istl using the new Communication interface in an Schwarz method
-->Nils-Arne DreierNils-Arne Dreier