#1538 Handling skipped tests
|Reported by||Christoph Grüninger (email@example.com)|
|Reported at||Dec 2, 2014 12:41|
|Operating System||Unspecified / All|
|Last edited by||Carsten Gräser (firstname.lastname@example.org)|
|Last edited at||May 6, 2015 12:01|
In Autotools one can indicate a skipped test with the magic return code 77. This is useful, when a external dependency like MPI or Alberta is not found or the the current environment does not support a feature i.e. range-based for loops. This is used in dune-common for parallel tests. In dune-grid many tests are disabled with Automake conditionals when the external grids are not found.
CMake 3.0 gained the feature to interpret a return code as a skipped test, using the test property SKIP_RETURN_CODE.
How should we treat skippable tests in the future?
- A) Let the build system exclude them, people can virtually not find out how many tests were skipped.
- B) Let them fail.
- C) Mimic with CMake the behavior of Autotools and let all tests return 77 if they are not run. For CMake 3.0 or newer there is a way to mark the tests as skipped. For CMake 2.8.12 and older..
- Ci) ..we can treat them as errors (like currently in dune-common).
- Cii) ..exclude them silently (like currently in dune-grid).