#1538 Handling skipped tests
Metadata
| Property | Value |
|---|---|
| Reported by | Christoph Grüninger (gruenich@iws.uni-stuttgart.de) |
| Reported at | Dec 2, 2014 12:41 |
| Type | Discussion |
| Version | 2.3 |
| Operating System | Unspecified / All |
| Last edited by | Carsten Gräser (graeser@math.fu-berlin.de) |
| Last edited at | May 6, 2015 12:01 |
Description
In Autotools one can indicate a skipped test with the magic return code 77. This is useful, when a external dependency like MPI or Alberta is not found or the the current environment does not support a feature i.e. range-based for loops. This is used in dune-common for parallel tests. In dune-grid many tests are disabled with Automake conditionals when the external grids are not found.
CMake 3.0 gained the feature to interpret a return code as a skipped test, using the test property SKIP_RETURN_CODE.
How should we treat skippable tests in the future?
- A) Let the build system exclude them, people can virtually not find out how many tests were skipped.
- B) Let them fail.
- C) Mimic with CMake the behavior of Autotools and let all tests return 77 if they are not run. For CMake 3.0 or newer there is a way to mark the tests as skipped. For CMake 2.8.12 and older..
- Ci) ..we can treat them as errors (like currently in dune-common).
- Cii) ..exclude them silently (like currently in dune-grid).