Skip to content

General policy on when to skip and when to omit tests

We currently have no general policy on when a test is to be skipped through the 77-mechanism and when it should be omitted from the test suite. The implementation usually differs in the following way:

  • You omit the test in CMakeLists by guarding it:
if (MYPACKAGE_FOUND)
  dune_add_test(... mytest ...)
endif()
  • You write your executable like this
#if HAVE_MYPACKAGE
 ...
#else
  return 77;
#endif

The first one has the disadvantage that a user that has not installed MYPACKAGE skips the associated tests silently. This is even more severe when people think they have the dependency installed, but it was not correctly detected. Then again, the latter bears the disadvantage that it clutters the implementation of the actual test quite a bit.

The purpose of this issue is to discuss the following points:

  • Can we settle for one of the above to be the Dune way?
  • Which one would that be?
  • Is the following alternative approach also interesing to you?

In dune-testtools we need to apply a mechanism that integrates the guarding variables into the actual test macro. This could as well be integrated into dune_add_test, such that this snippet:

if(UG_FOUND)
  dune_add_test(SOURCES testug.cc)
endif()

turns into

dune_add_test(SOURCES testug.cc
              CMAKE_GUARD UG_FOUND)

The latter approach could, if UG was not found switch the source to a dummy file that just returns 77 and thus achieving visibility of the entire testing suite.