Skip to content

Handle DUNE dependencies at find_package

Santiago Ospina De Los Ríos requested to merge feature/cmake-find-dependencies-on-config-file into master

Description

Currently, the dune dependencies are handled during the dune_project call by calling dune_create_dependency_tree(). This means that projects that want to use dune modules need to be dune modules. This is a very strong restriction because downstream users may not want to handle things in the build system as we do in DUNE.

Proposal

Move the dependency handling of the packages at find_package time so that all of the dependencies are resolved after this call. It can be achieved by setting up the CMake config file with the appropriated local dependencies. This will allow users to have very minimal contact with the dune build system if that's what they want.

find_package(dune-grid REQUIRED)
target_link_libraries(my_target PUBLIC Dune::Grid) # no need of `dune_project()` in order to use dune-grid

In this MR we limit ourselves to resolve DUNE dependencies. Additional module requirements can be stated individually by each module with the DUNE_CUSTOM_PKG_CONFIG_SECTION CMake variable.

Note: this goal is not quite possible after merging this because include directories are not properly set up #332, config files still depend on the dune build system !1262 (merged), and other packages are not yet resolved. However, this is another step in this direction.

Requires

Technical details

Role of ${ProjectName}-config.cmake

This is essentially an CMake script run when a CMake package is found. Since dune-common provides this CMake config file at finalize_dune_project(), no changes need to be done to downstream modules as long as they follow the dune_project()/finalize_dune_project() pattern and do not provide a custom cmake/pkg/${ProjectName}-confing.cmake.in. So this change is very minimal for dune modules. However, modules that deviated from the dune build system and do not call dune_project()/finalize_dune_project() or generate a custom CMake config file will need to provide their direct dependency resolution by themselves. I would argue that any project doing this was relying on implementation details of dune-common, or already solve their dependency tree without dune-common help.

Order of module resolution

The order of the module resolution is changed in this MR. Modules are currently found in the order of ALL_DEPENDENCIES created at dune_create_dependency_tree(), whereas this MR gets this behavior inverted: we find the required modules and add them to the ALL_DEPENDENCIES variable. If we see a downstream bug, it's likely that the module relies on the module order resolution. Or because some function/macro in dune-common relies on a given order. I think that it could be possible to reproduce the same ordering, but I would only invest time into this if a bizarre cascading bug appears in a downstream module. So we need to test with the whole stack before merging this.

The beauty of using the CMake config file properly is that dependencies are resolved individually by each module to guarantee that upstream and downstream dependencies are locally resolvable. In other words, single module dependencies are transitive to downstream modules. In this form, the first module to request an upstream module will be the only one to actually trigger its configuration. This means that downstream projects (theoretically) do not need to care anymore about the dependency tree (see Role of ALL_DEPENDENCIES below).

Current tests show that for modules of the same version altering this order resolution (f36b33fa) does not matter:

Exported upstream library resolution

In order to consume the dune targets, the targets dependencies need to be resolved at find_package(<lib>) as well. This also applies for upstream libraries like TBB, pthreads, MPI, etc. There are several options for this and is matter of discussion at the moment, see !1249 (comment 127761).

  • Call all the dune macros
  • Resolve found packages via pkg-config
  • Register dependencies into targets, and resolve dependencies on exported targets.

This this discussion will be moved to another MR.

Package Config

The PkgConfig mode is completely disabled here. In theory, downstream modules will not find module dependencies via PkgConfig anymore. But if I understand this correctly, this never worked properly anyways. See !910 and !1249 (comment 127724)

Versioning

It turns out that CMake version handling is incompatible with Dune's: !958. So this MR keeps the status quo, that is, find any instances of the dependent or suggested module and check that the version found is compatible with the one requested.

Backward compatibility

Modules using an older version of dune-common will still find packages at dune_project call, even if intermediate modules are producing the updated configuration files that are able to resolve dependencies at find_package calls.

Forward compatibility

Because we are generating config files for downstream projects, variables, functions and macros within these configurations files need to be public and stable over different dune-common releases. In this case, that is:

  • dune_check_module_version
  • DUNE_FOUND_DEPENDENCIES

Role of ALL_DEPENDENCIES

The function dune_create_dependency_tree builds a variable names ALL_DEPENDENCIES which contains all dune dependencies, and each downstream module seems to operate on ALL_DEPENDENCIES through dune-common macros/functions. This MR still builds an equivalent variable DUNE_FOUND_DEPENDENCIES in a different order and makes it available to the configured module so that everything works as before (it's renamed because the name is not internal anymore as mentioned in Forward compatibility section). To be clear, this MR does not aim to remove this variable, but ideally, such traversal over all modules needs to be removed and transmit information exclusively via the config file (i.e. ALL_DEPENDENCIES/DUNE_FOUND_DEPENDENCIES does not exists and module requirements are transmitted from direct dependent modules rather than the whole stack). This would simplify build logic because information is always local and only a function on the direct dependencies. As a discussion for future improvements, I found this variable being used in the following scenarios:

  • Generation of config.h (will be removed in the future due to !1262 (merged))
  • Python bindings metadata and requirements
  • Set up the venv path if not provided (use first upstream module with a venv if any)
  • Include all python module paths to the venv.
Edited by Santiago Ospina De Los Ríos

Merge request reports