WIP: Fix pkg config

Improve generation of pkg-config files. This allows (or will allow) to use DUNE without the DUNE build-system and in particular without cmake, just using the information provided by pkg-config.

See #188

Feature

• both dune- and external-dependencies are added to dune-common.pc
• cmake helper functions assist in collecting these dependencies
• for external dependecies we either use their existing pc fiels or write our own ones (containing the information extracted from the cmake target)
• helper functions write the pc files to keep the necessary effort in other modules as small as possible (see dune-geometry!155 and dune-istl!416)
• several external dependencies are already fully supported

Possible Features

The current changes offer the possibility to use the pkg-config files in DUNE modules further down the stack and avoid rerunning test already performed in one of the dependencies.

Open Questions

• there are still some packages that a tricky to implement. In particular suitesparse still needs some discussion.
• we might transition several tests to use pkg-config and by this reduce the cmake code we have to maintain.
• how do we properly handle config.h, although this is a general question and not specific to pkg-config.

Using DUNE without cmake

With these new features it is now possible to use DUNE without the DUNE build-system and in particular without cmake, just using the information provided by pkg-config.

An example can be found in infrastructure/pkg-config-test, which will serve us as test-infrastructure.

The user has to set the PKG_CONFIG_PATH and can then find the necessary DUNE flags.

> export PKG_CONFIG_PATH="[INSTALLDIR]/lib/pkg-config:[INSTALLDIR]/lib/pkg-config/dune:$PKG_CONFIG_PATH"

The he/she can use basically any type of build-system.

Here follow three examples:

Plain Makefile

CXXFLAGS = -g -Wall $(shell pkg-config --cflags dune-istl)
LDFLAGS = -g -Wall $(shell pkg-config --libs-only-L dune-istl)
LDLIBS = $(shell pkg-config --libs-only-l --libs-only-other dune-istl)

CC=g++
CXX=g++

all: myduneprogram

Meson

project('myduneprogram', 'cpp')

dune_istl = dependency('dune-istl')
executable('myduneprogram', ['src/myduneprogram.cc'], dependencies: dune_istl)

Plain CMake

cmake_minimum_required(VERSION 3.13)
project(myduneprogram LANGUAGES CXX)

include(FindPkgConfig)
pkg_check_modules(dune_istl REQUIRED IMPORTED_TARGET dune-istl)

add_executable(myduneprogram src/myduneprogram.cc)
target_link_libraries(myduneprogram PRIVATE PkgConfig::dune_istl)
target_include_directories(myduneprogam PRIVATE ${CMAKE_BINARY_DIR})

mentioned in issue #188

Ok, I get your general idea. Maybe we get it working. However, it might be necessary to do things slightly different:

• I don't think we should add individual library flags and includes and libs directly to the .pc file (like you have done for Vc), but provide a similar function and variable as for the pkf-config-requirements also for flags, include-dirs, libs and private-libs.
• The whole registration of the variable must be local to the module, i.e., ${dune-common_PKG_CONFIG_REQUIREMENTS} since all the configuration are run in all downstream modules again and thus we would clutter this variable with all the upstream requirements of all module dependencies.
• The pkg-config requirements/flags/include-dirs/libs can be added only if the module uses/links-against the corresponding package. Including the file add_dune_<package>_flags only provides a macro to link against it, but does not do the linking. E.g. Metis is provided in dune-common but not used at all, only for a test.
• I'm not sure if there is a unified way to find out whether a package is used by a module and must be linked. Most external dependencies are not hard dependencies, but just allow to use a library. If not used, it is not required to link against it. Thus, essentially, most external dependencies might not be required in the .pc file and the user is responsible for providing it, but some external dependencies are explicitly used and must be linked. This needs to be distinguished.

added 1 commit

• 25dbcfe1 - [pkg-config] check that list is non-empty before checking for duplicates

Compare with previous version

added 1 commit

• f1342e6b - [pkg-config] prepend is not available in older cmake versions

Compare with previous version

added 1 commit

• 7f93ce3e - add cmake function dune_create_and_install_pkg_config for packages

Compare with previous version

added 1 commit

• 5c7c14ac - Add pkg-config files for several external packages

Compare with previous version

added 1 commit

• dfcdd3f9 - Add HAVE_FOO flags

Compare with previous version

added 1 commit

• 5fd11ed7 - move HAVE_FOO flag to dune-common.pc file and check for existing pkg-config file

Compare with previous version

Some packages are really complicated to handle, e.g. suitesparse. It is defined with multiple (optional) components and external dependencies. So, the find-package file should be adapted to export everything not just as targets, but also as lists and variables.

Another problem is, that each dune module could search for a different version of a package, theoretically. E.g. in dune-common we search for parmetis >= 4.0 whereas in dune-istl any version is search for. On the other hand, in dune-common we first seach for METIS in any version, and indirectly again within the FindParMETIS for version METIS >= 3.0. Coditionally, we search for ptscotch, it scotchmetis is found instead of standard METIS. This requires to also generate a .pc file for scotch. If parmetis find ptscotchparmetis it searches for ptscotch that also requires scotch.

Since each module again might include the addPackageFlags.cmake files, the corresponding .pc files are generated multiple times.

I'm not sure that our current approach of generating auxilary pc-files is the right one.

Some of the problems you mentioned (mixed versions etc.) could be considered a bug in DUNE, as we surely don't want to mix versions and simply don't have any means to prohibit this.

Perhaps we should discuss the different options in a conference call with whoever is interested in this topic.

Each dune module will generate .pc files again. But this is not actually a problem, I think. If all these pc files have the same content than each of them could be used with pkg-config

added 1 commit

• e0634f01 - Update several pc descriptions and the generation from tagrets

Compare with previous version

@simon.praetorius thanks for all the effort. I have the impression that we are on a good route, but there are some things where we have to decide on one out of several solutions. It would be good to discuss the different options and come to a decision before going too deep into the rabbit hole.

If a package defines multiple targets, e.g. SuiteSparse::SuiteSparse and SuiteSparse::UMFPACK, how to extract a pkg-config file from this? Either create for each sub-target a separate pkg-config file, or merge the libraries and flags from all these sub-targets? It might be that some other packages depend on just a sub-target of e.g. suitesparse. Then, what to do? Recursively create a pkg-config file for just this sub-target and all its dependencies, or try to figure out the corresponding master target SuiteSparse::SuiteSparse that contains everything.

Another problem I have observed is that it is not easily possible to evaluate generator-expressions. Some packages set flags or include-directories or link-libraries including generator expressions, e.g. MPI or QuadMath. While in the case of QuadMath I can simply modify our cmake files, I do not have access to the corresponding find module for MPI. If we generate for all dependent libraries/targets a pkg-config file, we would also generate one for MPI This includes the flags "$<$<COMPILE_LANGUAGE:CUDA>:SHELL:-Xcompiler >". I don't want to write an interpreter/converter for this.

Creating pkg-config files with a "dune" prefix is tricky. When to add the prefix and when not? If we create for ALL libraries a pkg-config file, we could add this prefix always. Otherwise, we need to distinguish fo which libraries we have to add the prefix and for which not.

Since the Dune<module>Macros.cmake files are recursively included in all downstream modules, all the Add<package>Flags.cmake files are included as well, resulting in creating pc files not only multiple times, but also adding the requirement from all upstream modules to all downstream modules. E.g. gmp is not only a dependency to dune-common but also to dune-istl although it is not explicitly requested there. The underlying problem is, that all Macro files contain the find_package() calls and the include(Add<package>Flags) statements. This is a general problem I wanted to cleanup within cmake.

added 1 commit

• 2fd05567 - Cleanup the implementation of the pkg-config files

Compare with previous version

added 1 commit

• 290d2a72 - fix install directory

Compare with previous version

The pkg-config can be used (currently) only if the dune-package is installed, i.e., not from within the build-directory. This is because of the current way of setting include directories in the .pc files

I started a small project https://gitlab.dune-project.org/infrastructure/pkg-config-test to the current state of our pkg-config support.

mentioned in merge request dune-istl!416

mentioned in merge request dune-geometry!155

added 2 commits

• 6c366136 - [pkg-config] only write manual pc files, if they don't exist globally
• aeb7b9b5 - [pkg-config] add a comment to the template, warnign that this file should not...

Compare with previous version

added 1 commit

• 9e7589f2 - [pkg-config] install auxiliary pc files to sub directory

Compare with previous version

added 1 commit

• 1acc6cdf - Cleanup the pkg-config generation

Compare with previous version

added 1 commit

• 2ddf4e25 - fix typo in template.pc.in file

Compare with previous version

changed the description

added 1 commit

• c57bd39e - fix version string transformation for pkg-config

Compare with previous version

added 1 commit

• bad9d88c - default pkg-config name to filebase

Compare with previous version

mentioned in merge request infrastructure/dune-nightly-test!1 (closed)

Can we perhaps get this on the agenda again? I still think pkg-config support would be a major improvement.

mentioned in merge request !1249 (merged)

added buildsystem label

Just for the record: the cmake guys where also working on a way to export target configurations in .pc files:

• https://gitlab.kitware.com/cmake/cmake/-/merge_requests/6363
• https://gitlab.kitware.com/cmake/cmake/-/issues/22621

But they failed. Maybe there is a new attempts that I missed.

mentioned in issue #360

mentioned in merge request !1394 (merged)

mentioned in issue #379 (closed)

mentioned in merge request !1501 (merged)