dune-testtools issueshttps://gitlab.dune-project.org/quality/dune-testtools/-/issues2022-03-18T07:46:54Zhttps://gitlab.dune-project.org/quality/dune-testtools/-/issues/142Using installed testtools2022-03-18T07:46:54ZRené FritzeUsing installed testtoolsFor basically the first time ever I'm now in a situation where I want to build a single module (MYMODULE) against a whole tree of dune modules that I've previously installed (into /usr/local).
MYMODULE uses testtools for parametrization...For basically the first time ever I'm now in a situation where I want to build a single module (MYMODULE) against a whole tree of dune modules that I've previously installed (into /usr/local).
MYMODULE uses testtools for parametrization. Its configure now fails because `dune-testtools/cmake/modules/DuneSystemtests.cmake` wants to
```
dune_execute_process(COMMAND ${CMAKE_BINARY_DIR}/run-in-dune-env dune_extract_static.py
--ini ${STATVAR_INIFILE}
--file ${CMAKE_CURRENT_BINARY_DIR}/interface.log
WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
ERROR_MESSAGE "Error extracting static info from ${STATVAR_INIFILE}")
```
which fails since `dune_extract_static.py` is not in the virtual environment.
It's also nowhere in `/usr/local`. Not using the virtual environment is also not an option, since testtools requires `DUNE_PYTHON_VIRTUALENV_SETUP=True`, right?https://gitlab.dune-project.org/quality/dune-testtools/-/issues/140export generated tests on cmake2021-05-10T14:19:44ZSantiago Ospina De Los Ríossospinar@gmail.comexport generated tests on cmake### Description
The names of the generated tests are not available after calling `dune_add_system_test` and `add_system_test_per_target` . Because of this, one has two options to modify properties of the resulting tests:
* Hand-write t...### Description
The names of the generated tests are not available after calling `dune_add_system_test` and `add_system_test_per_target` . Because of this, one has two options to modify properties of the resulting tests:
* Hand-write the test names in CMake.
* Reconstruct all possible names from the mini file.
The shortcomings of the two options are obvious: A lot of duplicated code and information for something that `dune_add_system_test` and `add_system_test_per_target` compute anyways.
### Proposal
Two options arise here:
1. Export the list of tests into a return variable.
2. Attach tests to a property of the created targets `CREATED_TARGETS`.
The second case does not work on aliased targets (see !140). Thus, I would go for the first one. Any other opinions on this?https://gitlab.dune-project.org/quality/dune-testtools/-/issues/137INIFILE parameter for dune_add_system_test is required but not checked for2020-10-26T10:55:34ZDominic Kempfdominic.kempf@iwr.uni-heidelberg.deINIFILE parameter for dune_add_system_test is required but not checked forThings fail pretty weirdly if one omits the `INIFILE` parameter from the call. We should check for its presence and give a meaningful error message if missing.Things fail pretty weirdly if one omits the `INIFILE` parameter from the call. We should check for its presence and give a meaningful error message if missing.https://gitlab.dune-project.org/quality/dune-testtools/-/issues/132Add comparison of parallel vtu (pvtu) with sequential vtu2020-10-21T16:59:00ZTimo KochAdd comparison of parallel vtu (pvtu) with sequential vtuI have a working implementation in Dumux that compares a pvtu file with a vtu file by combining all the pieces into one element tree by appending and takes care of the double vertices that thereby occur.
I'll try to bring that to dune-t...I have a working implementation in Dumux that compares a pvtu file with a vtu file by combining all the pieces into one element tree by appending and takes care of the double vertices that thereby occur.
I'll try to bring that to dune-testtools soon.Timo KochTimo Kochhttps://gitlab.dune-project.org/quality/dune-testtools/-/issues/127Normal ini files cant be used for system tests unless __name is specified2019-05-22T14:09:46ZDominic Kempfdominic.kempf@iwr.uni-heidelberg.deNormal ini files cant be used for system tests unless __name is specifiedIni files are a subset of meta ini files and as such should be valid during the whole tool chain. The `__name` key needs to default to the filename in that case.
cc @rhessIni files are a subset of meta ini files and as such should be valid during the whole tool chain. The `__name` key needs to default to the filename in that case.
cc @rhessDominic Kempfdominic.kempf@iwr.uni-heidelberg.deDominic Kempfdominic.kempf@iwr.uni-heidelberg.dehttps://gitlab.dune-project.org/quality/dune-testtools/-/issues/126Unify keys for IniGridFactory2018-01-10T09:32:15ZDominic Kempfdominic.kempf@iwr.uni-heidelberg.deUnify keys for IniGridFactoryCurrently, there is a mixup of keys depending on the grid implementation, e.g. some use `cells`, some `elements`. That kind of defeats the purpose of the factory.Currently, there is a mixup of keys depending on the grid implementation, e.g. some use `cells`, some `elements`. That kind of defeats the purpose of the factory.https://gitlab.dune-project.org/quality/dune-testtools/-/issues/124Performance testing - Evaluate the deal.ii approach2017-10-15T13:05:33ZDominic Kempfdominic.kempf@iwr.uni-heidelberg.dePerformance testing - Evaluate the deal.ii approachI talked to Timo Heister from the deal.ii team at SIAM CSE17 and he developed a new tackle on performance testing. We should have a look and decide whether we want to follow the same route.
Here is the summary:
* Instead of measuring ti...I talked to Timo Heister from the deal.ii team at SIAM CSE17 and he developed a new tackle on performance testing. We should have a look and decide whether we want to follow the same route.
Here is the summary:
* Instead of measuring times, which is a noisy, clunky procedure, they start measuring *instruction counts*.
* Instruction counts are deterministic, so they can be compared with very low tolerance values
* `valgrind --tool=callgrind` and some additional gymnastics do the job
* The code has to be instrumented with macros starting and stopping the measurements (the counting only really makes sense in innermost loops, like assembly, iterative solvers etc.)
* deal.ii uses this on a monthly basis with `git-bisect` (which is a nice alternative to CI)
* Timo has uploaded his code to [github](https://github.com/tjhei/callgrind-wrapper)https://gitlab.dune-project.org/quality/dune-testtools/-/issues/120VTK comparison for files written in `appendedraw` mode2023-02-24T18:22:30ZTimo KochVTK comparison for files written in `appendedraw` modeAdjust parser to also parse the appendedraw style binary data formatAdjust parser to also parse the appendedraw style binary data formathttps://gitlab.dune-project.org/quality/dune-testtools/-/issues/116meaning of absolute criterion in outputtreecompare2017-10-15T13:05:33ZAndreas Nüßingmeaning of absolute criterion in outputtreecompareI'm using the `outputtreecompare` with the `releases/2.4` branch and stumbled upon the 'absolute' criterion for comparing floats. In compareini.py, the values are compared using `abs(value-ini2[key]) < absolute * max(value, ini2[key])` (...I'm using the `outputtreecompare` with the `releases/2.4` branch and stumbled upon the 'absolute' criterion for comparing floats. In compareini.py, the values are compared using `abs(value-ini2[key]) < absolute * max(value, ini2[key])` (line 156) and I am wondering about the meaning of the `max` term. I would have expected the behaviour without this term (consider for example the case `value<=0` and `ini2[key]<=0`). Is this intended, and if so, is there a possibility to compare using `abs(value-ini2[key]) < absolute`?https://gitlab.dune-project.org/quality/dune-testtools/-/issues/115Add a matrix/vector diff testing tool to dune-testtools2020-11-17T14:21:54ZDominic Kempfdominic.kempf@iwr.uni-heidelberg.deAdd a matrix/vector diff testing tool to dune-testtoolsI need a testtool that does fuzzy comparison of sparse matrices.
I will implement this, probably along these lines:
* use `MatrixMarket` from dune-istl as output/reference format
* provide something equivalent for vectors
* Provide...I need a testtool that does fuzzy comparison of sparse matrices.
I will implement this, probably along these lines:
* use `MatrixMarket` from dune-istl as output/reference format
* provide something equivalent for vectors
* Provide convenience C++ code for the output in dune-testtools
* Do fuzzy comparison in the python wrapper
Any input to this work in progress is welcome in this issue.Dominic Kempfdominic.kempf@iwr.uni-heidelberg.deDominic Kempfdominic.kempf@iwr.uni-heidelberg.de