The current build system is implemented in CMake and was designed to replace the former system based on Autotools. Especially the modules, variable passing, and dependency handling is bad, as every module has to re-run the tests from all direct and indirect dependencys
Designs
Child items
...
Show closed items
Linked items
0
Link issues together to show that they're related.
Learn more.
I don't see an easy way out. But it could be done, even step by step.
This are the steps we should go:
Use CMakeCache.txt from dependencies to avoid re-running checks in
later modules. We need to add some extra sorting, as there is not always
a unique ordering. Take dune-common, dune-geometry and dune-istl. When
dune-geometry is run prior to -istl, dune-istl must use -geometry's
cache and not -common's. Dune-grid has to pick up -istl and so on.
Easiest solution would be alphabetical sorting of names to get a unique
ordering.
After this step, dune modules would know the CMake variables in later
modules.
Create individual config.h files, for dune-foo this could be
dune-foo-config.h. For every dependency dune-bar dune-foo-config.h would
include dune-bar-config.h. We can stick to config.h, which includes
dune-foo-config.h.
After this step, dune modules know the C preprocessor macros.
Improve handling of linking libraries. CMake offers private and
public interface propagation.
After this step, linking and passing additional arguments would be more
CMake-ish.
Fix everything that I currently forgot. Get rid of many Dune-ism,
that one wouldn't write today, that I am even not aware of.
If others have different ideas, please share them!
Is this sequence of steps
something that people agree upon?
Use CMakeCache.txt from dependencies to avoid re-running checks in
later modules.
As a cmake non-expert I am surprised by this. Are the CMakeCache.txt files
really intended to be used like this from third-party software?
You yourself pointed me to those config-file packages, and I wonder why
we cannot use those? AFAIU these would solve a separate problem: I have
gotten reports from people that want to use Dune modules like just another
library. This fails, because Dune modules don't export the relevant information.
We need to add some extra sorting, as there is not always
a unique ordering. Take dune-common, dune-geometry and dune-istl. When
dune-geometry is run prior to -istl, dune-istl must use -geometry's
cache and not -common's.
Hmm, wouldn't I just merge the two CMakeCache.txt files and discard
duplicate information?
I also don't think we should touch the CMakeCache.txt. This is a CMake generated file. It's content might be up to changes when cmake version changes. It's internal. So don't merge or rewrite or generate anything like this manually.
There is some name irritation in the last mails. You speak about config-files. Do you mean the generated config.h file, or do you mean the PackageConfig.cmake files? The latter should be the way to go. In those config file, the configuration of a module is layed down, similar to a FindPackage.cmake file. It introduces imported targets, the package dependencies and could even set variables of cmake-checks performed in that module (like, if we check for c++ features, this could be done in the dune-common module and then the result be packed into the DuneCommonConfig.cmake file + the annotation that it should not be re-checked again)
we should work more with imported and exported targets and target properties. External packages should be put as find_dependency in the corresponding DuneModuleConfig.cmake file
If a package is found in module A, it must be a required dependency in module B that depends on module A. This can be set in the corresponding PackageConfig file of module A. If a module was not found in module A, it should not be search for in module B. So, instead of including the ModuleMacros.cmake file that searchs for everything again, we should just put everything in the generated PackageConfig.cmake file that can control whether to search for a module, whether it is required and so on.
Some of the points above are already implemented in the dune cmake system. Some need improvement. Maybe we can first cleanup the generated DuneModuleConfig.cmake files and see what could be exported there and imported in the find_package(dune-module) calles in dependent modules.
Should we create some kind of working group to go into the details of the cmake system? How could this be organized? I think, sending messages here to the mailinglist is a good starting point, but makes productive discussion complicated. Maybe some topic specific Matrix/Slack/Mattermost/... chat room? Or a GitLab Issue in the dune-common repo?
There is some name irritation in the last mails. You speak about config-files. Do you mean the generated config.h file, or do you mean the PackageConfig.cmake files?
the latter. Sorry, I am not sure what their official name is.
Should we create some kind of working group to go into the details of the cmake system? How could this be organized?
How about we open a meta-bug for dune-common and chart out a roadmap there.
Then assign individual steps to volunteers (provides these exist) and discuss
technical issues of the individual steps in separate gitlab issues.
I like the idea of using a CMakeConfig file instead of the CMake cache. I was afraid of crating and maintaining the files. But overall it seems like the better solution.
Thanks @gruenich for opening this issue and for including the mailinglist history. I have created a repository to try out some ideas we discuss here, e.g. using the package-config.cmake files to set additional variables so that running some command in all modules is not necessary. One initial observation: if an output variable is already set, some commands are not run again. Example:
set(OUTPUT_VAR 1)check_cxx_source_compiles("..." OUTPUT_VAR)# not run again, because OUTPUT_VAR is already set
The repository https://gitlab.dune-project.org/simon.praetorius/dune-cmake just contains very basic files and is intended to mimic the dune modules directory structure to a minimum, while not making everything too complicated. I have not included any Dune cmake modules and probably all the install-directories are not yet correct, but it already works.
All FindPackage.cmake modules should introduce an imported target containing all the properties like include directories and link flags etc. When linked to a dune module that creates an exported target, the next find_package(dune-module) also checks whether these transitive targets are exported properly.
There is a common pattern for the FindPackage.cmake modules, illustrated in the FindHYPRE.cmake file in the repository above. Essentially it is something like:
if(HYPRE_FOUND AND NOT TARGET HYPRE::HYPRE)add_library(HYPRE::HYPRE INTERFACE IMPORTED)set_target_properties(HYPRE::HYPRE PROPERTIES INTERFACE_INCLUDE_DIRECTORIES "${HYPRE_INCLUDE_DIR}" INTERFACE_LINK_LIBRARIES "${HYPRE_LIBRARIES}")endif()
and then you just write
find_package(HYPRE [REQUIRED])target_link_libraries(dune-common PUBLIC HYPRE::HYPRE)
where the namespace HYPRE:: style is recommended to distingish a link target from a raw link library name and also illustrates that the target is an imported target containing all the information.
Unfortunately transitive package dependencies are not (yet) well/directly supported by cmake - it does not provide a special command for this. See also the stackoverflow question (without any answer)
The problem is: one has to list a dependency twice: 1. in the CMakeLists file as find_package and 2. in the PackageConfig file as find_dependency. The proposal in the thread above is to use a variable to collect all target dependency and then expand this list in the config file.
If module A introduces an optional dependencies X and if found links (PUBLIC) to that dependency, and module B requires module A, then also X is a required dependency of module B
Example:
dune-common:
find_package(HYPRE)if(HYBPRE_FOUND)target_link_libraries(dune-common PUBLIC HYPRE::HYPRE)endif(HYPRE_FOUND)
dune-geometry:
find_package(dune-common REQUIRED)target_link_libraries(dune-geometry PUBLIC Dune::dune-common)
Then HYPRE is a REQUIRED dependency in dune-geometry, since dune-common is a required dependency and links against HYPRE
unfortunately the opposite is not true. If HYPRE was not found in dune-common and thus dune-common not linked against HYPRE it nevertheless is a required dependency in dune-geometry. This needs to be fixed.
I'm not a DUNE developer myself but have had some experience with CMake. In our group, we had a similar (but much smaller) issue, where we have one base project Utopia which we wanted to use in several downstream modules, possibly without the need of installing the upstream package. We achieved this in plain CMake, and I hope I can help you achieve a similar thing in DUNE. Having worked with DUNE for quite some time the build system bugs me a bit and I want to help improve it.
(This will be a bit lengthy...)
For starters: Targets
Modern CMake revolves around the definition of targets. These are a complete set of instructions for compiling and linking an executable or library. The target properties can be defined publicly (required for linking) and privately (required for compiling). If a downstream module wants to use a library, it must only receive its target with the public properties and is ready to go. CMake intends to pass targets via Package Configuration files (see CMake Package Layout for details). Downstream modules ideally only include this file. Writing a FindPackage.cmake module for finding the upstream module is only required if the upstream module does not write a Package Config file. Writing your own find module is much more error prone, as you might miss some important requirements or specifications of the upstream module.
What is not "Modern CMake" in DUNE?
I see the DUNE build system in its current state as a hybrid of the old configure build system and modern CMake.
dunecontrol messes with CMake search paths (as far as I can see). As a rule of thumb, CMake works best with as little intrusion as possible.
dune.module file defines dependencies. The whole purpose of CMake is to check dependencies, so users should do that in the CMake way. DUNE modules should be included in downstream projects via the regular find_package(), and nothing else. dune.module also defines the version of the project it is contained in. The version is usually set in CMake via the project() command and then propagated via package version files.
config.h propagates pre-processor definitions. These can be attached to targets via target_compile_definitions() and will be made available when the target is linked through target_link_libraries().
target_link_libraries() is not enough to use DUNE libraries. Users also have to call dune_project(), dune_enable_all_packages(), finalize_dune_project(), and add #include "config.h" in their source files (see above).
My proposal
Raise required CMake version as high as possible: CMake only really gets good in recent versions. Even CMake 3.10 (available on Ubuntu 18.04 LTS) still has some serious shortcomings. I would go with the CMake developers here and say that CMake can be easily compiled from source with any C++ compiler, which should always be possible if a recent version is not available via a package manager. I know that DUNE is a bit conservative with its dependencies but I would urge you to reconsider this in order to have a clean and modern build system. CMake 3.12+ would be ideal.
Express everything needed downstream in targets: Every DUNE library defines its own library target with all required information. This entails include directories, pre-processor macros, etc. Linking an executable with a DUNE library then just works via target_link_library(my_exe PUBLIC DUNE::common). Importantly, this gets rid of a "list" of DUNE_LIBRARIES because requirements are propagated. For example, using DUNE-PDELab will just be target_link_library(my_exe PUBLIC DUNE::PDELab), and as PDELab is linked to DUNE::Common, ISTL, and so on, these are automatically linked to the final executable.
Write CMake Package Config files: DUNE modules write Package config filesDuneModuleConfig.cmake which ensure that the module can be used downstream and that the targets are available. In many ways, this config file mirrors the DuneModuleMacros.cmake module, but it only checks the CMake configuration for the package when used downstream. If this is done corrently, downstream modules do not need to write their own FindDuneModule.cmake modules.
I see that these files are already written automatically by the build system. However, they do not serve their intended purpose because they do not check for the module dependencies and users have no way of manipulating them.
Leave finding the modules to CMake: Users of DUNE should ideally just include the desired DUNE modules via CMake commands like find_package(dune-common 2.6 REQUIRED) or find_package(dune-alugrid). If specific package locations must be passed, users set a cache variable when calling CMake, like cmake -Ddune-common_ROOT=<path> ..
Use CMake Package Registry: One important feature of the DUNE build system is that the modules can be used from the build tree and that installing them is not necessary. This also works in "plain" CMake: The Package Config files can also be written into the build tree. For finding these packages, CMake provides a Package Registry which is bound to the user building the packages. Calls to find_package() inspect this registry and can find registered modules in non-standard paths without the need of setting a _ROOT variable.
Where to go from here?
The DUNE build system provides a lot of custom CMake functions to relieve module developers from writing CMake, which in itself is a nice thing. But having a proper build system entails that developers have to interact with it. It can be very helpful to have proper guidelines and a well documented example. In the long run, this might be more effective than trying to "save" developers from getting into CMake more deeply.
More concretely, I can offer to adopt what I envisage in dune-common and one downstream module for you to see what it may look like. But I figure that there is quite a number of things we would have to discuss beforehand. Let me know what you think!
The dunecontrol script is useful if building the whole tree of dune-modules. There it also determines the order to build the individual modules and then forwards the package dirs to the cmake commands. Maybe the same could be obtained by having something like a super-module based on cmake, with sub-packages that are automatically configured and build in order of their dependencies.
I see. The build order of the separate modules is indeed something that is not clear to users. But CMake keeps track of all internal dependencies and we could use that:
The easiest way of managing dependencies with CMake is indeed to include them into the project directory. CMake natively supports that new (sub-) projects are added via add_subdirectory(), and merges them into the build tree. We could create a "super"-repository which includes DUNE modules as subdirectories (users could place them there themselves or they could be included as submodules). The project would mainly contain a top-level CMakeLists.txt file which includes all subdirectories with a globbing expression.
This would have the advantage that CMake would always know how to build a certain target. In the build directory, users could call make dune-pdelab and CMake would detect the dependencies and compile all modules required for the PDELab library in one build.
On the downside, globbing is not recommended for source files because CMake might miss changes or additions to these. So the approach could be error-prone. Also, merging projects into a super-project might be confusing to users.
I'm not sure how much people want to stick to the dunecontrol script. Right now, it takes the key objective and feature of managing build and usage dependencies away from CMake, at least for dependencies between separate DUNE modules.
We had (or have (?)) also some issues #173 with version requirements specified in dune-module file vs. cmake files. They were (are) not compatible, completely. Just having one place where to sets the versions in just one style would probably resolve this issue easily
The package registry is also something I thought about. But, I'm not sure whether it works smoothly with the requirement that I want to be able to choose which dune installation to consider in my build, e.g. if I have multiple dune version installed (2.6 and 2.7 and git) or if I have multiple configurations of the same version installed (2.7 with mpi and 2.7 without mpi or one directory containing a bugfix). I want to have the control which one cmake chooses at the end, not just by the version number in find_package.
The package registry is very simple and only works via project names, so you can only have one entry per project at a time. If you have different builds of the same project, you can hint which one should be preferably used by find_package via the Package_ROOT variables, like so: