Core Modules issueshttps://gitlab.dune-project.org/groups/core/-/issues2023-09-22T14:15:23Zhttps://gitlab.dune-project.org/core/dune-common/-/issues/346ParallelError in dune-grid Python bindings with OpenMPI2023-09-22T14:15:23ZValentina SchüllerParallelError in dune-grid Python bindings with OpenMPIHi everyone,
I encountered an issue with the DUNE Python bindings last week, which I can trigger by calling `dune.grid.structuredGrid()`.
To summarize: I have to call `from mpi4py import MPI` before using functions from `dune.grid`, oth...Hi everyone,
I encountered an issue with the DUNE Python bindings last week, which I can trigger by calling `dune.grid.structuredGrid()`.
To summarize: I have to call `from mpi4py import MPI` before using functions from `dune.grid`, otherwise DUNE throws a `ParallelError`:
```py
import dune.grid
# from mpi4py import MPI <-- this line will prevent the RuntimeError
dune.grid.structuredGrid([-1], [0], [10])
```
```
RuntimeError: ParallelError [Communication:/home/valentina/mambaforge/envs/richards-swe/include/dune/common/parallel/mpicommunication.hh:118]:
You must call MPIHelper::instance(argc,argv) in your main() function before using the MPI Communication!
```
It seems like MPI is not initialized properly by the Python bindings in this setup.
**Remark 1.** The following additional import statements _do not_ prevent this error:
- `import mpi4py`
- `import dune.common`
- it also does not help to do `from dune.grid import structuredGrid`
**Remark 2.** If I use MPICH instead of OpenMPI, I do not need `from mpi4py import MPI`. Checking the [DUNE source code for the error](https://github.com/dune-project/dune-common/blob/master/dune/common/parallel/mpicommunication.hh#L118) indicates that there is a difference between OpenMPI and MPICH with respect to how/if MPI is initialized by DUNE/the Python bindings?
**Remark 3.** Note that I never actually use `MPI` from mpi4py in this example. A linter will thus complain about the import statement, which is unfortunate.
## Environment
Latest version of DUNE modules installed via PyPI (`pip install dune-fem`):
```
dune-alugrid 2.9.0.2
dune-common 2.9.0
dune-fem 2.9.0.2
dune-geometry 2.9.0
dune-grid 2.9.0
dune-istl 2.9.0
dune-localfunctions 2.9.0
...
mpi4py 3.1.4
```
- Python: 3.10.11 (inside a mamba/conda environment)
- gcc/g++ 11.3 (Ubuntu 22.04)
- OpenMPI: 4.1.2.
I verified that OpenMPI and mpi4py are installed and working correctly on my system, using a [simple MPI example](https://mpitutorial.com/tutorials/mpi-hello-world/) and the [first example from the mpi4py tutorial](https://mpi4py.readthedocs.io/en/stable/tutorial.html#point-to-point-communication).
Let me know if you need any additional information!
Best,
ValentinaAndreas DednerAndreas Dednerhttps://gitlab.dune-project.org/core/dune-grid/-/issues/173MultipleCodimMultipleGeomTypeMapper not assignable2023-09-07T09:56:57ZChristian EngwerMultipleCodimMultipleGeomTypeMapper not assignableUntil 2017 the mapper was assignable. Later the `layout_` was changed to be `const`, which broke the (default) assignment.
Is this in purpose?
In general I'd consider assignability as something natural for this type of data. The fix wo...Until 2017 the mapper was assignable. Later the `layout_` was changed to be `const`, which broke the (default) assignment.
Is this in purpose?
In general I'd consider assignability as something natural for this type of data. The fix would be relatively simple.
@core: comments?https://gitlab.dune-project.org/core/dune-localfunctions/-/issues/18Installs an empty directory 'comm' into /usr/local/share/doc/dune-localfunctions2023-08-30T14:24:56ZYuri VicInstalls an empty directory 'comm' into /usr/local/share/doc/dune-localfunctionshttps://gitlab.dune-project.org/core/dune-localfunctions/-/issues/17Error during build: No rule to make target2023-08-30T14:03:56ZYuri VicError during build: No rule to make target```
gmake[6]: Entering directory '/usr/ports/math/dune-localfunctions/work/.build'
cd /usr/ports/math/dune-localfunctions/work/.build && /usr/local/bin/cmake -E cmake_depends "Unix Makefiles" /usr/ports/math/dune-localfunctions/work/dune...```
gmake[6]: Entering directory '/usr/ports/math/dune-localfunctions/work/.build'
cd /usr/ports/math/dune-localfunctions/work/.build && /usr/local/bin/cmake -E cmake_depends "Unix Makefiles" /usr/ports/math/dune-localfunctions/work/dune-localfunctions-4def7a7c627172d660854411d322818115a20765-4def7a7c627172d660854411d322818115a20765 /usr/ports/math/dune-localfunctions/work/dune-localfunctions-4def7a7c627172d660854411d322818115a20765-4def7a7c627172d660854411d322818115a20765/doc /usr/ports/math/dune-localfunctions/work/.build /usr/ports/math/dune-localfunctions/work/.build/doc /usr/ports/math/dune-localfunctions/work/.build/doc/CMakeFiles/install_dune-localfunctions-manual.pdf.dir/DependInfo.cmake --color=
Dependee "/usr/ports/math/dune-localfunctions/work/.build/doc/CMakeFiles/install_dune-localfunctions-manual.pdf.dir/DependInfo.cmake" is newer than depender "/usr/ports/math/dune-localfunctions/work/.build/doc/CMakeFiles/install_dune-localfunctions-manual.pdf.dir/depend.internal".
Dependee "/usr/ports/math/dune-localfunctions/work/.build/doc/CMakeFiles/CMakeDirectoryInformation.cmake" is newer than depender "/usr/ports/math/dune-localfunctions/work/.build/doc/CMakeFiles/install_dune-localfunctions-manual.pdf.dir/depend.internal".
Scanning dependencies of target install_dune-localfunctions-manual.pdf
gmake[6]: Leaving directory '/usr/ports/math/dune-localfunctions/work/.build'
/usr/local/bin/gmake -f doc/CMakeFiles/install_dune-localfunctions-manual.pdf.dir/build.make doc/CMakeFiles/install_dune-localfunctions-manual.pdf.dir/build
gmake[6]: Entering directory '/usr/ports/math/dune-localfunctions/work/.build'
gmake[6]: *** No rule to make target '/usr/ports/math/dune-localfunctions/work/dune-localfunctions-4def7a7c627172d660854411d322818115a20765-4def7a7c627172d660854411d322818115a20765/doc/dune-localfunctions-manual_safepdf', needed by 'doc/CMakeFiles/install_dune-localfunctions-manual.pdf'. Stop.
gmake[6]: Leaving directory '/usr/ports/math/dune-localfunctions/work/.build'
gmake[5]: *** [CMakeFiles/Makefile2:2384: doc/CMakeFiles/install_dune-localfunctions-manual.pdf.dir/all] Error 2
gmake[5]: Leaving directory '/usr/ports/math/dune-localfunctions/work/.build'
gmake[4]: *** [CMakeFiles/Makefile2:2391: doc/CMakeFiles/install_dune-localfunctions-manual.pdf.dir/rule] Error 2
gmake[4]: Leaving directory '/usr/ports/math/dune-localfunctions/work/.build'
gmake[3]: *** [Makefile:772: install_dune-localfunctions-manual.pdf] Error 2
```
But in the end the build succeeds.https://gitlab.dune-project.org/core/dune-localfunctions/-/issues/21FTBFS with gcc-132023-08-30T13:48:04ZOliver Sanderoliver.sander@tu-dresden.deFTBFS with gcc-13Reported by Debian for the 2.9.0 release: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1037634Reported by Debian for the 2.9.0 release: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1037634https://gitlab.dune-project.org/core/dune-common/-/issues/351[Python bindings] Compilation fails with Python 3.112023-08-30T11:01:21ZValentina Schüller[Python bindings] Compilation fails with Python 3.11Installing `dune-common` fails when using Python 3.11, seemingly due to issues in the `pybind11` code. The error messages I get are very similar to those reported [here](https://github.com/sirfz/tesserocr/issues/298) and fixing it is see...Installing `dune-common` fails when using Python 3.11, seemingly due to issues in the `pybind11` code. The error messages I get are very similar to those reported [here](https://github.com/sirfz/tesserocr/issues/298) and fixing it is seemingly related to using a newer version of cython which supports Python 3.11. E.g.,
```plaintext
FAILED: python/dune/typeregistry/CMakeFiles/_typeregistry.dir/_typeregistry.cc.o
/usr/bin/g++-12 -DENABLE_GMP=1 -DENABLE_MPI=1 -DENABLE_QUADMATH=1 -DHAVE_CONFIG_H -D_GLIBCXX_USE_FLOAT128 -D_typeregistry_EXPORTS -I/tmp/pip-install-enwkyj4j/dune-common_30d26d5c16504f4cadc30ff0c9fb6446/_skbuild/linux-x86_64-3.11/cmake-build -I/tmp/pip-install-enwkyj4j/dune-common_30d26d5c16504f4cadc30ff0c9fb6446 -I/home/valentina/mambaforge/envs/richards-swe/include/python3.11 -std=c++17 -O3 -DNDEBUG -O3 -DNDEBUG -fPIC -fext-numeric-literals -MD -MT python/dune/typeregistry/CMakeFiles/_typeregistry.dir/_typeregistry.cc.o -MF python/dune/typeregistry/CMakeFiles/_typeregistry.dir/_typeregistry.cc.o.d -o python/dune/typeregistry/CMakeFiles/_typeregistry.dir/_typeregistry.cc.o -c /tmp/pip-install-enwkyj4j/dune-common_30d26d5c16504f4cadc30ff0c9fb6446/python/dune/typeregistry/_typeregistry.cc
In file included from /tmp/pip-install-enwkyj4j/dune-common_30d26d5c16504f4cadc30ff0c9fb6446/dune/python/pybind11/attr.h:13,
from /tmp/pip-install-enwkyj4j/dune-common_30d26d5c16504f4cadc30ff0c9fb6446/dune/python/pybind11/pybind11.h:45,
from /tmp/pip-install-enwkyj4j/dune-common_30d26d5c16504f4cadc30ff0c9fb6446/dune/python/common/typeregistry.hh:21,
from /tmp/pip-install-enwkyj4j/dune-common_30d26d5c16504f4cadc30ff0c9fb6446/python/dune/typeregistry/_typeregistry.cc:8:
/tmp/pip-install-enwkyj4j/dune-common_30d26d5c16504f4cadc30ff0c9fb6446/dune/python/pybind11/cast.h: In function ‘std::string pybind11::detail::error_string()’:
/tmp/pip-install-enwkyj4j/dune-common_30d26d5c16504f4cadc30ff0c9fb6446/dune/python/pybind11/cast.h:446:36: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
446 | " " + handle(frame->f_code->co_filename).cast<std::string>() +
```
_See the attached file for the full error output:_ [dune_common_output.txt](/uploads/1fded00175fb83a485405c650a753bbf/dune_common_output.txt)
**Note:** This issue will most probably apply to `dune-grid`, `dune-fem`, etc. I have decided to open it here since this seems to be the baseline package for all of the others.
Here is some information about my system:
- `g++-12 --version`: 12.3.0
- `python3 --version`: Python 3.11.4
Full conda environment:
```plaintext
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
ca-certificates 2023.7.22 hbcca054_0 conda-forge
ld_impl_linux-64 2.40 h41732ed_0 conda-forge
libexpat 2.5.0 hcb278e6_1 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc-ng 13.1.0 he5830b7_0 conda-forge
libgomp 13.1.0 he5830b7_0 conda-forge
libnsl 2.0.0 h7f98852_0 conda-forge
libsqlite 3.42.0 h2797004_0 conda-forge
libuuid 2.38.1 h0b41bf4_0 conda-forge
libzlib 1.2.13 hd590300_5 conda-forge
ncurses 6.4 hcb278e6_0 conda-forge
openssl 3.1.2 hd590300_0 conda-forge
pip 23.2.1 pyhd8ed1ab_0 conda-forge
python 3.11.4 hab00c5b_0_cpython conda-forge
readline 8.2 h8228510_1 conda-forge
setuptools 68.1.2 pyhd8ed1ab_0 conda-forge
tk 8.6.12 h27826a3_0 conda-forge
tzdata 2023c h71feb2d_0 conda-forge
wheel 0.41.2 pyhd8ed1ab_0 conda-forge
xz 5.2.6 h166bdaf_0 conda-forge
```
Let me know if you need any additional information!
Best,
Valentinahttps://gitlab.dune-project.org/core/dune-common/-/issues/347dune_add_test does not work with aliased targets2023-08-01T16:03:30ZSantiago Ospina De Los Ríossospinar@gmail.comdune_add_test does not work with aliased targetsIt turns out that non-mutable targets are not usable with `dune_add_test` because the dune testing facilities want to mutate the target.
* With the introduction of !1247, we want to make more regular usage of aliased targets as they rep...It turns out that non-mutable targets are not usable with `dune_add_test` because the dune testing facilities want to mutate the target.
* With the introduction of !1247, we want to make more regular usage of aliased targets as they represent the non-mutable version of a target.
* Another instance of this case is when using an imported library or executable. Since they are imported, they are non-mutable.
The offending line is [here](https://gitlab.dune-project.org/core/dune-common/-/blob/48a5251d7f79d0203b7b20933b8589386212a8b9/cmake/modules/DuneTestMacros.cmake#L362), where the test wants to modify the properties of the target to (not) be excluded from `make all`.
```cmake
# Make sure to exclude the target from all, even when it is user-provided
if(DUNE_BUILD_TESTS_ON_MAKE_ALL AND (NOT ADDTEST_EXPECT_COMPILE_FAIL))
set_property(TARGET ${ADDTEST_TARGET} PROPERTY EXCLUDE_FROM_ALL 0)
else()
set_property(TARGET ${ADDTEST_TARGET} PROPERTY EXCLUDE_FROM_ALL 1)
endif()
```https://gitlab.dune-project.org/core/dune-grid/-/issues/172ParallelError in dune-grid Python bindings with OpenMPI2023-07-14T08:44:01ZValentina SchüllerParallelError in dune-grid Python bindings with OpenMPIHi everyone,
I encountered an issue with the DUNE Python bindings last week, which I can trigger by calling `dune.grid.structuredGrid()`.
To summarize: I have to call `from mpi4py import MPI` before using functions from `dune.grid`, oth...Hi everyone,
I encountered an issue with the DUNE Python bindings last week, which I can trigger by calling `dune.grid.structuredGrid()`.
To summarize: I have to call `from mpi4py import MPI` before using functions from `dune.grid`, otherwise DUNE throws a `ParallelError`:
```py
import dune.grid
# from mpi4py import MPI <-- this line will prevent the RuntimeError
dune.grid.structuredGrid([-1], [0], [10])
```
```
RuntimeError: ParallelError [Communication:/home/valentina/mambaforge/envs/richards-swe/include/dune/common/parallel/mpicommunication.hh:118]:
You must call MPIHelper::instance(argc,argv) in your main() function before using the MPI Communication!
```
It seems like MPI is not initialized properly by the Python bindings in this setup.
**Remark 1.** The following additional import statements _do not_ prevent this error:
- `import mpi4py`
- `import dune.common`
- it also does not help to do `from dune.grid import structuredGrid`
**Remark 2.** If I use MPICH instead of OpenMPI, I do not need `from mpi4py import MPI`. Checking the [DUNE source code for the error](https://github.com/dune-project/dune-common/blob/master/dune/common/parallel/mpicommunication.hh#L118) indicates that there is a difference between OpenMPI and MPICH with respect to how/if MPI is initialized by DUNE/the Python bindings?
**Remark 3.** Note that I never actually use `MPI` from mpi4py in this example. A linter will thus complain about the import statement, which is unfortunate.
## Environment
Latest version of DUNE modules installed via PyPI (`pip install dune-fem`):
```
dune-alugrid 2.9.0.2
dune-common 2.9.0
dune-fem 2.9.0.2
dune-geometry 2.9.0
dune-grid 2.9.0
dune-istl 2.9.0
dune-localfunctions 2.9.0
...
mpi4py 3.1.4
```
- Python: 3.10.11 (inside a mamba/conda environment)
- gcc/g++ 11.3 (Ubuntu 22.04)
- OpenMPI: 4.1.2.
I verified that OpenMPI and mpi4py are installed and working correctly on my system, using a [simple MPI example](https://mpitutorial.com/tutorials/mpi-hello-world/) and the [first example from the mpi4py tutorial](https://mpi4py.readthedocs.io/en/stable/tutorial.html#point-to-point-communication).
Let me know if you need any additional information!
Best,
Valentinahttps://gitlab.dune-project.org/core/dune-grid/-/issues/171Incomplete type `Dune::dgf::PeriodicFaceTransformationBlock::AffineTransforma...2023-07-13T09:30:32ZChristian HeinigkIncomplete type `Dune::dgf::PeriodicFaceTransformationBlock::AffineTransformation` with clang-15.0.7Hi all,
building the `releases/v2.9` and `master` branches with `clang 15.0.7` runs into the following error in an `ubuntu:lunar` container:
```
[ 47%] Building CXX object CMakeFiles/dunegrid.dir/dune/grid/io/file/dgfparser/blocks/ver...Hi all,
building the `releases/v2.9` and `master` branches with `clang 15.0.7` runs into the following error in an `ubuntu:lunar` container:
```
[ 47%] Building CXX object CMakeFiles/dunegrid.dir/dune/grid/io/file/dgfparser/blocks/vertex.cc.o
In file included from /dune/dune-grid/dune/grid/io/file/dgfparser/blocks/periodicfacetrans.cc:7:
In file included from /dune/dune-grid/dune/grid/io/file/dgfparser/blocks/periodicfacetrans.hh:9:
In file included from /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/vector:64:
/usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/stl_vector.h:1143:34: error: arithmetic on a pointer to an incomplete type 'Dune::dgf::PeriodicFaceTransformationBlock::AffineTransformation'
return *(this->_M_impl._M_start + __n);
~~~~~~~~~~~~~~~~~~~~~~ ^
/dune/dune-grid/dune/grid/io/file/dgfparser/blocks/periodicfacetrans.hh:44:32: note: in instantiation of member function 'std::vector<Dune::dgf::PeriodicFaceTransformationBlock::AffineTransformation>::operator[]' requested here
return transformations_[ i ];
^
/dune/dune-grid/dune/grid/io/file/dgfparser/blocks/periodicfacetrans.hh:29:14: note: forward declaration of 'Dune::dgf::PeriodicFaceTransformationBlock::AffineTransformation'
struct AffineTransformation;
^
In file included from /dune/dune-grid/dune/grid/io/file/dgfparser/blocks/periodicfacetrans.cc:7:
In file included from /dune/dune-grid/dune/grid/io/file/dgfparser/blocks/periodicfacetrans.hh:9:
In file included from /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/vector:64:
/usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/stl_vector.h:988:50: error: arithmetic on a pointer to an incomplete type 'Dune::dgf::PeriodicFaceTransformationBlock::AffineTransformation'
{ return size_type(this->_M_impl._M_finish - this->_M_impl._M_start); }
~~~~~~~~~~~~~~~~~~~~~~~ ^
/dune/dune-grid/dune/grid/io/file/dgfparser/blocks/periodicfacetrans.hh:49:33: note: in instantiation of member function 'std::vector<Dune::dgf::PeriodicFaceTransformationBlock::AffineTransformation>::size' requested here
return transformations_.size();
^
/dune/dune-grid/dune/grid/io/file/dgfparser/blocks/periodicfacetrans.hh:29:14: note: forward declaration of 'Dune::dgf::PeriodicFaceTransformationBlock::AffineTransformation'
struct AffineTransformation;
^
[ 52%] Building CXX object CMakeFiles/dunegrid.dir/dune/grid/io/file/dgfparser/dgfparser.cc.o
2 errors generated.
gmake[2]: *** [CMakeFiles/dunegrid.dir/build.make:188: CMakeFiles/dunegrid.dir/dune/grid/io/file/dgfparser/blocks/periodicfacetrans.cc.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
In file included from /dune/dune-grid/dune/grid/io/file/dgfparser/dgfparser.cc:14:
In file included from /dune/dune-geometry/dune/geometry/referenceelements.hh:14:
In file included from /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/vector:64:
/usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/stl_vector.h:1143:34: error: arithmetic on a pointer to an incomplete type 'Dune::dgf::PeriodicFaceTransformationBlock::AffineTransformation'
return *(this->_M_impl._M_start + __n);
~~~~~~~~~~~~~~~~~~~~~~ ^
/dune/dune-grid/dune/grid/io/file/dgfparser/blocks/periodicfacetrans.hh:44:32: note: in instantiation of member function 'std::vector<Dune::dgf::PeriodicFaceTransformationBlock::AffineTransformation>::operator[]' requested here
return transformations_[ i ];
^
/dune/dune-grid/dune/grid/io/file/dgfparser/blocks/periodicfacetrans.hh:29:14: note: forward declaration of 'Dune::dgf::PeriodicFaceTransformationBlock::AffineTransformation'
struct AffineTransformation;
^
In file included from /dune/dune-grid/dune/grid/io/file/dgfparser/dgfparser.cc:14:
In file included from /dune/dune-geometry/dune/geometry/referenceelements.hh:14:
In file included from /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/vector:64:
/usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/stl_vector.h:988:50: error: arithmetic on a pointer to an incomplete type 'Dune::dgf::PeriodicFaceTransformationBlock::AffineTransformation'
{ return size_type(this->_M_impl._M_finish - this->_M_impl._M_start); }
~~~~~~~~~~~~~~~~~~~~~~~ ^
/dune/dune-grid/dune/grid/io/file/dgfparser/blocks/periodicfacetrans.hh:49:33: note: in instantiation of member function 'std::vector<Dune::dgf::PeriodicFaceTransformationBlock::AffineTransformation>::size' requested here
return transformations_.size();
^
/dune/dune-grid/dune/grid/io/file/dgfparser/blocks/periodicfacetrans.hh:29:14: note: forward declaration of 'Dune::dgf::PeriodicFaceTransformationBlock::AffineTransformation'
struct AffineTransformation;
^
2 errors generated.
gmake[2]: *** [CMakeFiles/dunegrid.dir/build.make:258: CMakeFiles/dunegrid.dir/dune/grid/io/file/dgfparser/dgfparser.cc.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:1897: CMakeFiles/dunegrid.dir/all] Error 2
gmake: *** [Makefile:146: all] Error 2
--- Failed to build dune-grid ---
```
I'm resulting to an older image at the moment, because I have not much time to debug this. I just wanted to write this issue down somewhere and figured you might want to take a look.
Best,
Christianhttps://gitlab.dune-project.org/core/dune-common/-/issues/341FTBFS with gcc-132023-07-12T08:54:18ZOliver Sanderoliver.sander@tu-dresden.deFTBFS with gcc-13Reported by Debian for the 2.9.0 release: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1037630
```
cd /<<PKGBUILDDIR>>/build/dune/common/simd/test && /usr/bin/c++ -DENABLE_MPI=1 -DENABLE_QUADMATH=1 -DHAVE_CONFIG_H -D_GLIBCXX_USE_FL...Reported by Debian for the 2.9.0 release: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1037630
```
cd /<<PKGBUILDDIR>>/build/dune/common/simd/test && /usr/bin/c++ -DENABLE_MPI=1 -DENABLE_QUADMATH=1 -DHAVE_CONFIG_H -D_GLIBCXX_USE_FLOAT128 -I/<<PKGBUILDDIR>>/build -I/<<PKGBUILDDIR>> -isystem /usr/lib/x86_64-linux-gnu/openmpi/include -isystem /usr/lib/x86_64-linux-gnu/openmpi/include/openmpi -std=c++17 -g -O2 -ffile-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIE -fext-numeric-literals -MD -MT dune/common/simd/test/CMakeFiles/looptest.dir/looptest.cc.o -MF CMakeFiles/looptest.dir/looptest.cc.o.d -o CMakeFiles/looptest.dir/looptest.cc.o -c /<<PKGBUILDDIR>>/build/dune/common/simd/test/looptest.cc
In file included from /usr/include/c++/13/cassert:44,
from /<<PKGBUILDDIR>>/dune/common/simd/interface.hh:17,
from /<<PKGBUILDDIR>>/dune/common/simd/simd.hh:13,
from /<<PKGBUILDDIR>>/dune/common/simd/loop.hh:13,
from /<<PKGBUILDDIR>>/build/dune/common/simd/test/looptest.cc:12:
/<<PKGBUILDDIR>>/dune/common/simd/loop.hh: In constructor ‘Dune::LoopSIMD<T, S, A>::LoopSIMD()’:
/<<PKGBUILDDIR>>/dune/common/simd/loop.hh:70:31: error: ‘uintptr_t’ does not name a type
70 | assert(reinterpret_cast<uintptr_t>(this) % std::min(alignof(LoopSIMD<T,S,A>),alignof(std::max_align_t)) == 0);
| ^~~~~~~~~
/<<PKGBUILDDIR>>/dune/common/simd/loop.hh:14:1: note: ‘uintptr_t’ is defined in header ‘<cstdint>’; did you forget to ‘#include <cstdint>’?
13 | #include <dune/common/simd/simd.hh>
+++ |+#include <cstdint>
14 | #include <dune/common/typetraits.hh>
/<<PKGBUILDDIR>>/dune/common/simd/loop.hh: In constructor ‘Dune::LoopSIMD<T, S, A>::LoopSIMD(const Dune::LoopSIMD<T, S, OA>&)’:
/<<PKGBUILDDIR>>/dune/common/simd/loop.hh:82:31: error: ‘uintptr_t’ does not name a type
82 | assert(reinterpret_cast<uintptr_t>(this) % std::min(alignof(LoopSIMD<T,S,A>),alignof(std::max_align_t)) == 0);
| ^~~~~~~~~
/<<PKGBUILDDIR>>/dune/common/simd/loop.hh:82:31: note: ‘uintptr_t’ is defined in header ‘<cstdint>’; did you forget to ‘#include <cstdint>’?
make[5]: *** [dune/common/simd/test/CMakeFiles/looptest.dir/build.make:835: dune/common/simd/test/CMakeFiles/looptest.dir/looptest.cc.o] Error 1
make[5]: *** Waiting for unfinished jobs....
```https://gitlab.dune-project.org/core/dune-common/-/issues/337Libraries with NO_EXPORT option are exported as module libraries2023-07-04T11:44:33ZSantiago Ospina De Los Ríossospinar@gmail.comLibraries with NO_EXPORT option are exported as module librariesIf a library target asks to not be exported (i.e., `dune_add_library(libname NO_EXPORT [...])`) it gets excluded from the export set of the library (e.g. [this line](https://gitlab.dune-project.org/core/dune-common/-/blob/fa23cf444d632f9...If a library target asks to not be exported (i.e., `dune_add_library(libname NO_EXPORT [...])`) it gets excluded from the export set of the library (e.g. [this line](https://gitlab.dune-project.org/core/dune-common/-/blob/fa23cf444d632f9447292cc1a9eb07e88a2fce18/cmake/modules/DuneAddLibrary.cmake#L186)). That's the expected behavior.
However, the library _name_ is still exported in the `${${ProjecName}_LIBRARIES}` variable in the export file: set [here](https://gitlab.dune-project.org/core/dune-common/-/blob/fa23cf444d632f9447292cc1a9eb07e88a2fce18/cmake/modules/DuneAddLibrary.cmake#L192) and exported [here](https://gitlab.dune-project.org/core/dune-common/-/blob/fa23cf444d632f9447292cc1a9eb07e88a2fce18/cmake/modules/DuneProject.cmake#L199). This is not correct as the library was not exported and consumers have no idea how to resolve such a library name. In particular, this fails when the consumer wants to enable all dune packages with `dune_enable_all_packages`, `dune_target_enable_all_packages` or directly linking to the dependent libraries.
The correct behavior is incidentally fixed in !1247 by having internal and external library modules: `${${ProjecName}_INTERFACE_LIBRARIES}` is set [here](https://gitlab.dune-project.org/core/dune-common/-/merge_requests/1247/diffs#a18835a94745be4f5f1e2d86451897c5accab3d3_183_214) and exported [here](https://gitlab.dune-project.org/core/dune-common/-/merge_requests/1247/diffs#28f9da52000ccbac5f7d99b068322a2dae1d5a5d_200_204), where this variable is only set for exported module libraries.
_Edit: Link to the export set is updated to the correct line._Simon PraetoriusSimon Praetoriushttps://gitlab.dune-project.org/core/dune-common/-/issues/342`dune-py` configure failure in `dune-istl`2023-06-29T09:20:59ZAndreas Dedner`dune-py` configure failure in `dune-istl`This was discussed in a closed MR:
https://gitlab.dune-project.org/core/dune-common/-/merge_requests/1251#note_128976This was discussed in a closed MR:
https://gitlab.dune-project.org/core/dune-common/-/merge_requests/1251#note_128976https://gitlab.dune-project.org/core/dune-common/-/issues/340[Python] Former JIT compilation confuses latter2023-06-14T17:54:19ZAlexander Müller[Python] Former JIT compilation confuses latterI have a new grid implementation where I'm currently adding Python bindings.
Whereas I managed to register and compile my grid, later compilation failed, no matter what object.
The code at hand is:
```python
reader = (readeriga.json, "....I have a new grid implementation where I'm currently adding Python bindings.
Whereas I managed to register and compile my grid, later compilation failed, no matter what object.
The code at hand is:
```python
reader = (readeriga.json, "../../iga/test/auxiliaryFiles/element.ibra")
gridView = igaGrid(reader, dimgrid=2,dimworld=2)
v = FieldVector((2,1)) # this line fails
```
The error is as follows:
```python
Traceback (most recent call last):
File "/dune/dune-common/build-cmake/python/dune/common/__init__.py", line 86, in FieldVector
return globals()[fv](values)
KeyError: 'FieldVector_2'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/dune-iga/dune/python/test/readGrid.py", line 23, in <module>
v= FieldVector((2,1))
File "/dune/dune-common/build-cmake/python/dune/common/__init__.py", line 90, in FieldVector
cls = _loadVec(includes, typeName).FieldVector
File "/dune/dune-common/build-cmake/python/dune/common/__init__.py", line 74, in _loadVec
return generator.load(
File "/dune/dune-common/build-cmake/python/dune/generator/generator.py", line 172, in load
return self.post(moduleName, source, postscript, extraCMake)
File "/dune/dune-common/build-cmake/python/dune/generator/generator.py", line 124, in post
module = builder.load(moduleName, source, self.typeName[0], extraCMake)
File "/dune/dune-common/build-cmake/python/dune/generator/cmakebuilder.py", line 388, in load
module = importlib.import_module("dune.generated." + moduleName)
File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'dune.generated.fieldvector_2371cd404c10f9fc2031e51d552c58df'
```
A similar error is created for every other object I try to compile afterward. Thus, I think there is no issue regarding the FieldVector.
I researched and debugged the issue and during first problem arises in
[cmakebuilder.py685](https://gitlab.dune-project.org/core/dune-common/-/blob/releases/2.9/python/dune/generator/cmakebuilder.py?ref_type=heads#L685).
This process fails with error code -11 (I don't know what that means, if it is important, or even if it is always reproducible).
After this the [line](https://gitlab.dune-project.org/core/dune-common/-/blob/releases/2.9/python/dune/generator/cmakebuilder.py?ref_type=heads#L388) fails, since the module as never generated.
Interestingly, if I do this by hand and mimic the make command as follows:
```bash
/dune/dune-common/build-cmake/dune-env/.cache/dune-py/python/dune/generated# make -q -f CMakeFiles/fieldvector_2371cd404c10f9fc2031e51d552c58df.dir/fieldvector_2371cd404c10f9fc2031e51d552c58df.make fieldvector_2371cd404c10f9fc2031e51d552c58df.so
```
and
`bash buildScript.sh fieldvector_2371cd404c10f9fc2031e51d552c58df`
my Python script from above does not raise the exception and
```bash
DUNE-DEBUG: Module fieldvector_2371cd404c10f9fc2031e51d552c58df not loaded
DUNE-DEBUG: Loading fieldvector_2371cd404c10f9fc2031e51d552c58df
```
works as expected.
How can this behavior be explained? I refactored my code several times but no difference.https://gitlab.dune-project.org/core/dune-common/-/issues/327FindPython3 not working correctly.2023-05-24T12:15:27ZRobert KFindPython3 not working correctly.I have two Python versions installed, `3.10.9` and `3.11.1`.
The default for `python` or `python3` is `3.10.9`. For cmake > 3.20 the cmake tests find the `3.11.1` version leading to import errors later on. The old backport (used when c...I have two Python versions installed, `3.10.9` and `3.11.1`.
The default for `python` or `python3` is `3.10.9`. For cmake > 3.20 the cmake tests find the `3.11.1` version leading to import errors later on. The old backport (used when cmake < 3.20) finds the Python version correctly. Is there a way to make the new cmake test behave correctly?
Seems like I can solve the problem by manually specifying
```
-DPython3_EXECUTABLE=/usr/bin/python
```
But it's strange that this is not the first pick for the cmake test.https://gitlab.dune-project.org/core/dune-common/-/issues/223FMA optimization for LoopSIMD2023-05-07T20:47:05ZNils-Arne DreierFMA optimization for LoopSIMDThe FMA optimization of compilers fail for `LoopSIMD` expressions like
```
x += alpha*b;
```
if the length of the `LoopSIMD` exceeds a compiler dependent size. You can see it in [this Godbold example](https://godbolt.org/z/Ws5PPd). For g...The FMA optimization of compilers fail for `LoopSIMD` expressions like
```
x += alpha*b;
```
if the length of the `LoopSIMD` exceeds a compiler dependent size. You can see it in [this Godbold example](https://godbolt.org/z/Ws5PPd). For gcc-10 the critical size is 7 and for clang-11 it is 28.
The problem also occurs if one uses another SIMD types like `Vec8d` or `Vc::Vector<double>` in `LoopSIMD` to build a larger SIMD type.
I see three possible solutions:
1. Wait for better compilers (I do not know how likely that is)
2. Exchange the expressions with a function call like `x = fma(alpha,b,x)` that does the multiplication and addition interleaved. A fallback for arithmetic types are given by [`std::fma`](https://en.cppreference.com/w/cpp/numeric/math/fma). That would require to change a lot of code. However, I think the most important places are in `densematrix.hh` and `densevector.hh` as alot of other performance relevant things, like operator application and preconditions in dune-istl, build up on this.
3. We could return an expression template from the `operator*` that is evaluated when it is assigned to `LoopSIMD` and in the `operator +=` it interleaves the multiplication and addition such that the compiler can do the optimization.
I already made a first attempt for this: https://gitlab.dune-project.org/nils.dreier/dune-common/-/commits/LoopSIMD_fma_expression_template . But I think it would be difficult to convince the CI machinery for that.
For a SpMV operation in dune-istl, I can observe a speedup up to `2x`.
@joe I think you have the best overview over the SIMD things. Would be great to hear your opinion.https://gitlab.dune-project.org/core/dune-common/-/issues/336Python-related build-failure on the 2.9 branch2023-05-04T19:58:09ZOliver Sanderoliver.sander@tu-dresden.dePython-related build-failure on the 2.9 branchI cannot build the `releases/2.9` branch on my up-to-date Debian testing machine. After a fresh clone, I get
```
~/dune-2.9> ./dune-common/bin/dunecontrol all
--- going to build dune-common dune-geometry dune-localfunctions dune-uggrid...I cannot build the `releases/2.9` branch on my up-to-date Debian testing machine. After a fresh clone, I get
```
~/dune-2.9> ./dune-common/bin/dunecontrol all
--- going to build dune-common dune-geometry dune-localfunctions dune-uggrid dune-grid dune-istl dune-matrix-vector dune-solvers dune-typetree dune-gmsh4 dune-functions dune-fufem dune-elasticity dune-gfe ---
--- calling all for dune-common ---
--- calling vcsetup for dune-common ---
--- calling cmake for dune-common ---
cmake "/home/sander/dune-2.9/dune-common"
-- The C compiler identification is GNU 12.2.0
-- The CXX compiler identification is GNU 12.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test cxx_std_flag_17
-- Performing Test cxx_std_flag_17 - Success
-- Performing Test compiler_supports_cxx17
-- Performing Test compiler_supports_cxx17 - Success
-- Looking for std::experimental::make_array<int,int>
-- Looking for std::experimental::make_array<int,int> - found
-- Looking for std::move<std::experimental::detected_t<std::decay_t,int>>
-- Looking for std::move<std::experimental::detected_t<std::decay_t,int>> - found
-- Looking for std::identity
-- Looking for std::identity - not found
-- Performing Test DUNE_HAVE_CXX_UNEVALUATED_CONTEXT_LAMBDA
-- Performing Test DUNE_HAVE_CXX_UNEVALUATED_CONTEXT_LAMBDA - Failed
-- Found LATEX: /usr/bin/latex
-- Found LatexMk: /usr/bin/latexmk (found version "Version 4.79")
-- Could NOT find Sphinx (missing: SPHINX_EXECUTABLE)
-- Found Doxygen: /usr/bin/doxygen (found version "1.9.4") found components: doxygen dot
-- Found PkgConfig: /usr/bin/pkg-config (found version "1.8.1")
-- Performing tests for dune-common (from /home/sander/dune-2.9/dune-common/cmake/modules/DuneCommonMacros.cmake)
-- Set Minimal Debug Level to 4
-- Looking for sgemm_
-- Looking for sgemm_ - not found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Looking for dgemm_
-- Looking for dgemm_ - found
-- Found BLAS: /usr/lib/x86_64-linux-gnu/libblas.so;/usr/lib/x86_64-linux-gnu/libf77blas.so;/usr/lib/x86_64-linux-gnu/libatlas.so
-- Looking for cheev_
-- Looking for cheev_ - not found
-- Looking for cheev_
-- Looking for cheev_ - found
-- Found LAPACK: /usr/lib/x86_64-linux-gnu/liblapack.so;/usr/lib/x86_64-linux-gnu/libblas.so;/usr/lib/x86_64-linux-gnu/libf77blas.so;/usr/lib/x86_64-linux-gnu/libatlas.so
-- Looking for dsyev_
-- Looking for dsyev_ - found
-- Found GMP: /usr/lib/x86_64-linux-gnu/libgmpxx.so
-- Performing Test QuadMath_COMPILES
-- Performing Test QuadMath_COMPILES - Success
-- Found QuadMath: (Supported by compiler)
-- Found MPI_C: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so (found suitable version "3.1", minimum required is "3.0")
-- Found MPI: TRUE (found suitable version "3.1", minimum required is "3.0") found components: C
-- Could NOT find TBB (set TBB_DIR to path containing TBBConfig.cmake or set PKG_CONFIG_PATH to include the location of the tbb.pc file) (missing: PkgConfigTBB_LINK_LIBRARIES PkgConfigTBB_FOUND) (found version "")
-- Could NOT find PTScotch (missing: SCOTCH_LIBRARY SCOTCHERR_LIBRARY SCOTCH_INCLUDE_DIR)
-- Found METIS: /usr/lib/x86_64-linux-gnu/libmetis.so (found version "5.1")
-- Found METIS: /usr/lib/x86_64-linux-gnu/libmetis.so (found suitable version "5.1", minimum required is "5.0")
-- Found MPI_C: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so (found version "3.1")
-- Found MPI: TRUE (found version "3.1") found components: C
-- Found ParMETIS: /usr/lib/libparmetis.so (found suitable version "4.0", minimum required is "4.0")
-- Could NOT find Vc (missing: Vc_DIR)
-- Found Python3: /usr/bin/python3 (found version "3.11.2") found components: Interpreter Development Development.Module Development.Embed
-- Found pip_/usr/bin/python3: TRUE
-- Failed to find the python package virtualenv with interpreter /usr/bin/python3. (missing: DUNE_PYTHON_virtualenv_FOUND)
-- Found venv_/usr/bin/python3: TRUE
-- Building a virtualenv in /home/sander/dune-2.9/dune-common/build-cmake/dune-env
-- Found pip_/home/sander/dune-2.9/dune-common/build-cmake/dune-env/bin/python: TRUE
-- Using scripts from /home/sander/dune-2.9/dune-common/cmake/scripts for creating doxygen stuff.
-- using /home/sander/dune-2.9/dune-common/doc/doxygen/Doxystyle to create doxystyle file
-- using C macro definitions from /home/sander/dune-2.9/dune-common/doc/doxygen/doxygen-macros for Doxygen
-- Skipping building CMake API documentation (Sphinx was not found!)
-- Installing python package abstract requirements: jinja2 mpi4py numpy pip>=21.a portalocker setuptools>=41 wheel
-- Installing python package at /home/sander/dune-2.9/dune-common/build-cmake/python/. into Dune virtual environment
-- Generating the CMake metadata file at dune/data/dune-common.cmake
-- Not adding custom target for config.h generation
-- The following OPTIONAL packages have been found:
* LATEX
* LatexMk
* Doxygen, Class documentation generator, <www.doxygen.org>
To generate the class documentation from C++ sources
* BLAS, fast linear algebra routines
* LAPACK, fast linear algebra routines
* GMP, GNU multi-precision library, <https://gmplib.org>
* QuadMath, GCC Quad-Precision Math Library, <https://gcc.gnu.org/onlinedocs/libquadmath>
* Inkscape, converts SVG images, <www.inkscape.org>
To generate the documentation with LaTeX
* Threads, Multi-threading library
* METIS (required version >= 5.0), Serial Graph Partitioning, <http://glaros.dtc.umn.edu/gkhome/metis/metis/overview>
* MPI, Message Passing Interface library
Parallel programming on multiple processors
* ParMETIS (required version >= 4.0), Parallel Graph Partitioning, <http://glaros.dtc.umn.edu/gkhome/metis/parmetis/overview>
* Python3
-- The following OPTIONAL packages have not been found:
* Sphinx, Documentation generator, <www.sphinx-doc.org>
To generate the documentation from CMake and Python sources
* TBB, Intel's Threading Building Blocks, <https://github.com/oneapi-src/oneTBB>
* PTScotch, Sequential and Parallel Graph Partitioning, <https://gitlab.inria.fr/scotch/scotch>
* Vc, C++ Vectorization library, <https://github.com/VcDevel/Vc>
For use of SIMD instructions
-- Configuring done
CMake Warning (dev) in doc/comm/CMakeLists.txt:
Policy CMP0087 is not set: Install CODE|SCRIPT allow the use of generator
expressions. Run "cmake --help-policy CMP0087" for policy details. Use
the cmake_policy command to set the policy and suppress this warning.
This warning is for project developers. Use -Wno-dev to suppress it.
-- Generating done
-- Build files have been written to: /home/sander/dune-2.9/dune-common/build-cmake
--- calling make for dune-common ---
build directory: build-cmake
cmake --build . --
[ 0%] Building CXX object CMakeFiles/dunecommon.dir/dune/common/debugalign.cc.o
[ 20%] Building CXX object CMakeFiles/dunecommon.dir/dune/common/debugallocator.cc.o
[ 20%] Building CXX object CMakeFiles/dunecommon.dir/dune/common/exceptions.cc.o
[ 20%] Building CXX object CMakeFiles/dunecommon.dir/dune/common/fmatrixev.cc.o
[ 20%] Building CXX object CMakeFiles/dunecommon.dir/dune/common/ios_state.cc.o
[ 40%] Building CXX object CMakeFiles/dunecommon.dir/dune/common/parametertree.cc.o
[ 40%] Building CXX object CMakeFiles/dunecommon.dir/dune/common/parametertreeparser.cc.o
[ 40%] Building CXX object CMakeFiles/dunecommon.dir/dune/common/path.cc.o
[ 60%] Building CXX object CMakeFiles/dunecommon.dir/dune/common/simd/test.cc.o
[ 60%] Building CXX object CMakeFiles/dunecommon.dir/dune/common/stdstreams.cc.o
[ 60%] Building CXX object CMakeFiles/dunecommon.dir/dune/common/stdthread.cc.o
[ 80%] Linking CXX static library lib/libdunecommon.a
[ 80%] Built target dunecommon
[ 80%] Building CXX object python/dune/common/CMakeFiles/_common.dir/_common.cc.o
In file included from /home/sander/dune-2.9/dune-common/dune/python/pybind11/attr.h:13,
from /home/sander/dune-2.9/dune-common/dune/python/pybind11/pybind11.h:45,
from /home/sander/dune-2.9/dune-common/dune/python/common/typeregistry.hh:21,
from /home/sander/dune-2.9/dune-common/dune/python/common/dynmatrix.hh:15,
from /home/sander/dune-2.9/dune-common/python/dune/common/_common.cc:10:
/home/sander/dune-2.9/dune-common/dune/python/pybind11/cast.h: In function ‘std::string pybind11::detail::error_string()’:
/home/sander/dune-2.9/dune-common/dune/python/pybind11/cast.h:446:36: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
446 | " " + handle(frame->f_code->co_filename).cast<std::string>() +
|
```https://gitlab.dune-project.org/core/dune-common/-/issues/333TransposedMatrixWrapper does not cast properly into FieldMatrix or DenseMatrix.2023-04-21T10:10:56ZRobert KTransposedMatrixWrapper does not cast properly into FieldMatrix or DenseMatrix.The recent commit https://gitlab.dune-project.org/core/dune-grid/-/commit/b2695b3c7d169348d39f20899b64a584a667aa49 breaks the compliation of the dune-spgrid python bindings because the Jacobian is returned as transposedView of the Jacobi...The recent commit https://gitlab.dune-project.org/core/dune-grid/-/commit/b2695b3c7d169348d39f20899b64a584a667aa49 breaks the compliation of the dune-spgrid python bindings because the Jacobian is returned as transposedView of the JacobianTransposed which is not a FieldMatrix. The return transposedView does not properly cast into FieldMatrix which is the assumed return type in the Python bindings.https://gitlab.dune-project.org/core/dune-common/-/issues/280[python] requirements.txt, install_requires, and Python-requires2023-04-15T10:44:53ZTimo Koch[python] requirements.txt, install_requires, and Python-requiresIn `dune_python_install_package` a `requirements.txt` is created for each module. Usually, this file contains the development requirements of a package. How does this relate to what is stated in `Python-requires`? Is this the way to add ...In `dune_python_install_package` a `requirements.txt` is created for each module. Usually, this file contains the development requirements of a package. How does this relate to what is stated in `Python-requires`? Is this the way to add development requirements? And packaging requirements are then stated `pyproject.toml`?
Where would be the right place to add `pylint` and `flake8` in the current setup which are only needed during development?
__UPDATE:__ A `requirements.txt` file is no longer created (see !1062). This resolves the confusion. This means currently only "minimal" required dependencies can be stated (through dune.module's `Python-requires` argument which is propagated to `setup.py`'s `install_requires`, see https://packaging.python.org/discussions/install-requires-vs-requirements/ for what `install_requires` is good for). However, the current default mode when running `dunecontrol all` creates a developer environment with local internal virtual env and editable package installs. Hence, it would be nice to have a way to specify _development_ requirements (e.g. linters or testing frameworks). This is a job that is often accomplished by providing such packages in a `requirement.txt` file (see e.g. https://packaging.python.org/discussions/install-requires-vs-requirements/, https://github.com/yngvem/python-project-structure, https://gitlab.dune-project.org/samuel.burbulla/dune-mmesh/-/blob/master/requirements.txt).
__Idea/suggestion:__ if a `requirement.txt` is found in the configured Python dune module (in the build directory) next to `setup.py`, the `requirements.txt` is used. There are several possibilities what "used" means, the simplest is that `requirements.txt` is considered as `pip install -e -r requirements.txt` when doing an editable installation.https://gitlab.dune-project.org/core/dune-common/-/issues/278add function to add a Python package during configure time2023-04-09T14:00:31ZAndreas Dedneradd function to add a Python package during configure timeProblem
-------
See https://gitlab.dune-project.org/quality/dune-testtools/-/merge_requests/142:
the module `dune-testtool` contains a Python package that needs to be installed during configure time so that tests within the module are ru...Problem
-------
See https://gitlab.dune-project.org/quality/dune-testtools/-/merge_requests/142:
the module `dune-testtool` contains a Python package that needs to be installed during configure time so that tests within the module are run correctly.
Solution
--------
A possible solution is to add a cmake function that allows installation of a Python package during configure time (after the internal venv is setup).
Alternative: add a flag to the existing `install_python` function to move installation of the package from the end of the build to the configure phase.https://gitlab.dune-project.org/core/dune-common/-/issues/248packagemetadata.py broken.2023-04-09T13:57:11ZRobert Kpackagemetadata.py broken.@christi: patch df8d14ba6f3d253be85cdc97b2758df18ee70a73 breaks the pip packaging. Python dependencies are not listed correctly anymore. The Python-Requires keyword in dune.module is missing in the valid_entries list, which is easy to fi...@christi: patch df8d14ba6f3d253be85cdc97b2758df18ee70a73 breaks the pip packaging. Python dependencies are not listed correctly anymore. The Python-Requires keyword in dune.module is missing in the valid_entries list, which is easy to fix but also the regex match string needs to account for '-'.
Testing can be done in dune-common by calling dunepackaging.py --onlysdist
The packages listed in Python-Requires in dune.module should then appear in the generated pyproject.toml