Require more recent version of CMake

I propose to target 3.13. Then Ubunut LTS and Debian users are happy out of the box. I don't care much about RHEL. openSuse has a strange version policy as they use packages from SLE. It's easy to get newer packages for openSuse, so no worry.

changed the description

CMake 3.13 has several advantages over CMake 3.10:

target_link_libraries, target_compile_options, and other functions for adding target properties work for INTERFACE targets (targets that are not compiled, like ~~template-only libraries~~ header-only modules).
Policy CMP0074 defaults to NEW: find_package(), and functions find_path() and find_library() inside find modules automatically use <Package>_ROOT (environment) variables as hints when searching.
Update of Python support, find_package(Python).

Another nice feature of CMake 3.14 (not sure if up for debate):

install(TARGETS) does not require DESTINATION arguments for executables, and static or shared libraries.

Policy CMP0074 defaults to NEW: find_package(), and functions find_path() and find_library() inside find modules automatically use _ROOT (environment) variables as hints when searching.

Funny. I did spent hours to modify CMake scripts to get rid off *_ROOT and use the old CMake preferred way and NOW they change it. Seriously!

What has preference?

Sometimes its good to admin that one has be wrong… ROOT is just a hint. If the variable is set directly, the find* call is not performed.

The idea is great: All calls to find a package automatically use <Package>_ROOT variables to search for it. But the introduction of the policy was a real headache: If you have CMake >=3.12, and the minimum required CMake version is below that, the policy defaults to ignoring any <Package>_ROOT variables. So one always has to set the policy explicitly. With CMake >=3.12, everything works out comfortably.

I'm fine with cmake 3.13

changed the description

Since cmake 3.11 the version-files support SameMinorVersion for compatibility checks. This would be nice, since in dune we change the minor version for introducing relevant changes. The major version is only very rarely changed. This differs from the classical semantic versioning.

Since cmake 3.12 the c++ standard 20 is known. But the c++ 17 standard is known since 3.8

changed the description

Concerning Ubuntu: I think that at the moment (and until July 2020) users of the 18.04LTS are not encouraged to upgrade to 20.04LTS and have to force the upgrade. That might keep people from doing that. Of course its not too difficult to install by hand or force the upgrade, I guess.

3.13

mentioned in merge request !809 (merged)

Nobody objects to require CMake 3.13.

@robert.kloefkorn @christi Any objections?

Otherwise I am going to prepare merge requests like !809 (merged) for all core modules and start merge them.

Until July (or so) it is not recommended to upgrade Ubuntu 18.04 to 20.04 - the upgrade command doesn't work without extra flag forcing the upgrade. I'm wondering if waiting until July - at least with merging this?

I don't want to wait. Their is currently a momentum related to the use of a more recent CMake version.
Users of master have to upgrade their Ubunut earlier.

I can live with that - although I really don't like the argument. I have a lot of people working together with me, including PhD students, who have to work with master branches because I need newer feature in dune-fem for example. But those are not necessarily people who are happy to upgrade their whole operating system prematurely just to be able to use Dune. So core user are not all power users (that is even more true for the Python side of things, where it is getting difficult for me to make sure that something new / a bugfix that I add to core is correctly moved into dune-python 2.7.

As discussed above. Upgrading the whole OS is not necessary. Just include another PPA (or deb repository) for a more recent cmake will do the trick. You can easily get cmake 3.17 without touching the rest of the system. Or, just downloading the binary works fine, too.

As I said, from my side, go ahead

Just as a reference (because I currently look up how to do it for our ci docker images with ubuntu 18.04): there is the official kitware apt-repository https://apt.kitware.com/ that provides up-to-date cmake packages for several ubuntu distributions.

Just a data point if someone's having trouble: I currently don't want to upgrade my Ubuntu (don't want to take the chance of breaking anything while I have to do multiple video conferences per week) but downloading the CMake binary from their website and setting the CMAKE variable to the corresponding binary is almost trivial and works fine.

Just include another PPA (or deb repository)

what is not really an option, if your machine is centrally administrated.

I don't see why wee need to merge these changes already now into master. In particular as most of the changes are currently highly experimental in Simons new dune-cmake repo.

I can live with that - although I really don't like the argument. I have a lot of people working together with me including PhD students who have to work with master branches because I need newer feature in dune-fem for example. But those are not necessarily people who are happy to upgrade their whole operating system prematurely just to be able to use dune. So core user are not all power users (that is even more true for the Python side of things were it is getting difficult for me to make sure that something new or a bugfix that I add to core is correctly moved into dune-python 2.7.

I just remembered @dominic saying that with newer CMake versions it might be possible to extract from the build which flags where used? That would be really nice for the dune-py construction which could then be build without the user having to provide the flags again. Unfortunately I can't find the discussion I had with Dominic anymore about this. Does somebody know if this would work?

I'm not sure whether I understand you correctly. Do you want to extract the compiler-flags and link-libraries etc. from the dune package (as library target) we linked against in a down-stream module, or do you want to extract the flags from the build-directory of an upstream module somehow?

I would guess case one. This is possible and this is the way we want to have the new cmake build-system to work. There, no compile-flags or link-libraries etc. are collected manually and need to be derived again in a downstream module. Instead, all those flags are associated directly to a library-target of the dune-modules and those targets transport these flags to the downstream module and forward all configurations automatically, when linked against it. Additionally, One could extract flags from the target properties if needed explicitly for something else, like python.

@andreas.dedner Might have been Steffen? I don't think I ever said such a thing.

Might have been as I said I can't find the details of that discussion anymore... But perhaps cmake -LA run in all dune modules would do the trick for me

changed milestone to %New CMake Build-System

closed via merge request !809 (merged)

mentioned in commit 57bf1afe

I just sent an email to our user mailing list, to inform everybody:

the master of the Dune core modules require CMake 3.13, cf. #200 (closed), !809 (merged). For most of you, this should not be a problem. If you don't have a recent enough CMake you can

remain before yesterday's commits
get a package from backports or however it is called for your distribution
download CMake 3.13 binaries from https://cmake.org/download/ Then you have to re-run dunecontrol and can add the variable CMAKE= You can place the variable definition in front of your dunecontrol command or into your opts file

Please also update your modules to require CMake 3.13. The CMake code from dune-common will be executed with the policies from your module. This mead lead to subtle changes. I didn't find any in my tests. This could also occur in the CMake code of your module.

Why have we done this? Because it will help us with improving the CMake-based build system, see #199 (closed), and the way we find Python (!806 (merged)).

This cmake version upgrade is really problematic, to say it in nice words. It immediately rules out user that use ubuntu 18.04 that may not want to upgrade their cmake version. I would suggest to revert this patch. I also have collaborators that use ubuntu 18.04 and that now cannot work with me anymore or have to undergo serious trouble to upgrade their cmake version. Lets not forget that cmake and the buildsystem is not our main target area. "Working" is perfectly fine for a build system and that is what we had before this nonsense patch was introduced. Also, decisions that rule out popular linux distributions require a formal vote from all core maintainers (developers) and not just a "nobody was objecting, merging in 5 minutes".

@gruenich: Please revert this! It breaks DUNE on ubuntu 18.04

No.

I do not follow your suggestion reverting my change. The actual nonsense would be reverting this after a discussion period of more than three weeks, having it in master for four months, and several changes based on this patch. I explicitly pinged you to ask for your opinion.

If you insist, I can initiate a formal vote on this topic.

Although, there are ways to solve the problem of missing cmake 3.13 in ubuntu 18:04, see the comments below, if it stays a hard problem and all solution do not apply, we have to find an alternative solution. It is a valid request to ensure that dune can be build (easily) in most of the widely used linux distributions. But, I also do not want to revert this change. The new cmake version requirement opens the possibility to improve the buildsystem and makes it ready for new c++ standards and requirements. If necessary, we could discuss to go back to 3.10, though. It requires some patches of the current cmake files and some workarounds but would not prohibit the new developments.

I personally think installing a newer CMake version e.g. from binaries or snap is much easier than installing or compiling something with Dune. So if it would for example documented on the Dune webpage how to install a newer CMake version this should be easily possible for all Dune users.

I've no strong opinion on this, but as far as I know Ubuntu 18.04 does provide newer CMake versions via snap which is official part of ubuntu (cf. https://snapcraft.io/cmake).

@robert.kloefkorn I see your point. Ubuntu 18.04 is a big linux distribution used nowadays and its current default package manager installs a too old cmake (in its default configuration) for the now required cmake_minimal_version. And there are some distributions build on top of this 18.04 ubuntu. So, you might need to change something in your way of providing the requirements for using dune in this distribution. This is not nice and I stumbled across this issue, when installing Dune on 10s of systems.

But, it is not a big issue. Using apt with default sources is just one way of installing software in ubuntu. There is

apt using the official kitware package repository
snap packages (Canonical pushes to make this the standard source for their software)
~~flatpak packages (similar to snap, but for more distributions)~~
Binary packages from the kitware website
...

All these are for some the standard way of installing software, so one cannot in generally argue that ubuntu 18.04 and derived distributions do not provide cmake in version >= 3.13, but it may require to change our way of installing dependencies. Cmake somehow is just a dependency. It can be installed in user-mode and system-wide, and all these solutions above require just 1 click or 1 command. So, probably not "serious trouble". If you need more details on the sources above, I can give assistance. If there are other reasons that I do not see, why cmake cannot be updated with the above, let me know so that we can find a simple other solution.

Maybe we should update the documentation or an FAQ page directing to the possible sources for installing this build-system dependency cmake.

I'm on 18.04 and have been using the binary from cmake.org since this patch was merged and had no issues. Only thing one needs to do is download them and setting CMAKE="YOUR_PATH/bin/cmake" in your DUNE opts file. This is of course even distribution agnostic and you do not have to touch your system-wide cmake version if you do not want to.

Alternatively, I just tried to install a newer cmake with snap and also had no problems: apt remove cmake + snap install --classic cmake.

@simon.praetorius: Thanks for the detailed description. Could you add this to the "how to install DUNE documentation"?

I think we can rest assure knowing that all people who contribute to this chat know how to fix this problem on their machine. I'm using gentoo, so almost all my packages are compiled from source and if you ask me, lets make cmake 3.18.2 the required version since it works for me and thus it should work for everybody else too. And if not, just switch to gentoo and then it will work ;-)

But seriously, the user who recently switched from Windoze to Ubuntu to write his/her master thesis may be overwhelmed with this. Also, the same student who switched half a year ago when 20.04 was not released and is wrapping up his/her thesis right now does not want to do the distupgrade, because things can get ugly. Another student was wanting to try out DUNE but it even fails to build out-of-box, so that person just went 2 deal with something else...

For that reasons I think 3.10.2 (or whatever the standard is on 18.04) should be the required minimum version. And all cmake implementations should be worked around. Otherwise, those should be viewed as staging and/or experimental and not be in the master branch. I can't believe that the changes between 3.13 and 3.10 are that drastic. As for the new standard: parts of the core don't even compile with cxx20, neither gcc-10-20 nor clang-10-20. So maybe that should be fixed first? But it also shows that this build system "fix" is not desperately needed right now. And then the last questions would be: What is installed on the clusters in Jülich and in Dresden?

@gruenich: I think a vote might be the right thing to do, or would have been the right thing in the first place. You sure may have pinged me but I was awfully busy this summer and I may have easily overlooked the consequences. But from the chat above I saw that Andreas objected and in that case you can count this vote for two because we are on the same side on this one.

But it also shows that this build system "fix" is not desperately needed right now.

The current build system where every module has to find the same dependencies again and again is awfully slow. That is of course mostly for developers who run cmake on a regular basis, still it's a lot of wasted time.

I'm very greatful for the work in particular @simon.praetorius has put in to work towards a more modern build system version with exported and imported targets. As I understand a newer CMake version is required for that change and setting the bar low on the version will give us a suboptimal solution with workarounds that needs updating (work) again soon.

Also I think it's simply not possible to develop such a big feature on some development branch only. Who's going to review all that code? So the only option seems to be to split it up into small feature branches and merge incrementally. And master as unstable future version should be the right place, no?

Another student was wanting to try out DUNE but it even fails to build out-of-box, so that person just went 2 deal with something else...

If "how to install cmake 3.13" would be part of the installation instructions this wouldn't be a problem, right? Also arguably we should have checked out the current release (CMake >= 3.1), as described in the installation instructions to "try out DUNE". (Maybe the "Note:" could be more prominent.)

Otherwise, those should be viewed as staging and/or experimental and not be in the master branch

When I go on the documentation page https://www.dune-project.org/doxygen/ the master branch is marked as "unstable". To get a stable version working on ubuntu 18.04 you would install release 2.7, no? I don't know when there will be a release 2.8 but judging from the recent distances between releases that will take a while (releases are work, work force is limited). So I guess by the time of release 2.8 it will not be very likely anymore that people use Ubuntu 18.04LTS since 20.04LTS is out.

But seriously, the user who recently switched from Windoze to Ubuntu to write his/her master thesis may be overwhelmed with this. Also, the same student who switched half a year ago when 20.04 was not released and is wrapping up his/her thesis right now does not want to do the distupgrade, because > things can get ugly. Another student was wanting to try out DUNE but it even fails to build out-of-box, so that person just went 2 deal with something else...

That's what we have releases for. You are not seriously suggesting that such people should use the git master?

What is installed on the clusters in Jülich and in Dresden?

We have lots of stuff installed on Taurus, including the latest and greatest of everything. No problems here.

@oliver.sander: Sometimes it is necessary, because our release cycles are rather long. It should not be the norm though. However, assuming that we release 2.8 soon the issue will persist.

Concerning snap: I just tried it and the argument seems moot:

On my Ubuntu laptop you seem to need special permissions to install snap (e.g.sudo) as I need to trust the author of the package. I doubt that everybody has these and that security-savy admins would allow this in a corporate setting.
Of course I do have sudo right. Hence I tried, but it refuses because of security concerns and I have to pass --classic to force installation.
Still the system cmake is the default, cmake --version proves that. To get the new version, I need to adapt the path such that /snap/bin is prioritized.

I must admit that this is far from trivial, personally I would rather compile it myself (but I am not an average user.

--classic just mean that the installation will not be in a sandbox. This is what you always get with apt. Also you need sudo permissions for apt. Hence snap --classic is not much different to apt. This all boils down to the question if you trust the maintainer of the package (which you would not ask for stuff available with apt, although there's good reasons to do so in universe). For the cmake snap this is one of the cmake devs.

... and the ubuntu snap package is even listed on the official cmake download webpage.

My main point was, that you need sudo rights.

Just for reference, I want to add an additional method to get recent cmake: It is pip-installable. So python -m pip install cmake will give you a non-sudo installation (into ~/.local/bin) of the latest version of cmake.

Thanks for this hint. I will add this as another option for installing cmake in the install instructions faq on the website.

That sounds intriguing. Could we add that as a command in dunecontrol? We then could warn users and tell them how install cmake with dunecontrol, and only use the pip-installed one if the system installed one does not suffice.

@markus.blatt: Thanks for bringing this up. This was my main complaint earlier but it seemed that until now I was the only one that thought this was a problem.

@dominic is right, pip install cmake installs a standalone binary package of cmake version 3.18.2. I did it in a virtual env to not mess up other things. It is a much better workaround then this snap option.

In general people don't have sudo rights and are not able to install stuff system-wide. For good reasons. If you have sudo right, why not install a Deb/RPM package? Even the local installation is no problem, just pass the right path via CMAKE variable in your opts file.

I also don't like to add more complexity in dunecontrol. I would like to get it better maintainable and not adding complexity.

I have now updated the installation instructions: https://dune-project.org/doc/installation/ to give in an FAQ style some hints how to provide cmake and the compilers in the required versions

Thanks for putting the work in - greatly appreciated since was really a mess. I was two questions:

is there a way to add a toc to the top of the page. Since the troubleshooting guide way down people might not look that far down if they fail right at the beginning
concerning the config.opts file: what is the default optimization used when running dunecontrol as suggested right at the top without any options? I think we do are ourselves a disservice if we suggest to people playing around with the software to use a version that is not compiled with Release build type since not having -DNDEBUG and at -O2 really slows down the code in my experience and gives the wrong impression about the efficiency of Dune.

good point about the TOC. I will try to add this in the installation instruction. I also wanted to add a comment about alternative installation instructions, like deb files, docker images and so on. This is currently only in the beginners resources.

Regarding the default optimization: It follows the defaults in cmake, I think. This typically does not set any optimization flags. Theoretically, one could add an option to dunecontrol (e.g. -debug) to set debug mode and have something like RelWithDebInfo by default. But, as pointed out before, the current dunecontrol is already difficult to maintain, so it is questionable whether it should be added/changed. But it should be clearly communicated what is happening by default.

I would really like an easy and well documented way to tell people how to speedup there code, especially for new users thinking of dune as a replacement for their own code or some other packages. If they have difficulties figuring out how to switch to some sort of release build when comparing the efficiency of dune with some other software, they might conclude that dune is too slow. Here are a few options:

have ./dunecontrol build Release by default
add a --release flag to ./dunecontrol (that is a but like petsc does it with their --with-debugging=0 configure option
CMAKE_BUILD_TYPE is I think not read from environment by CMake? So CMAKE_BUILD_TYPE=Release ./dunecontrol all will not work. But we could make ./dunecontrol aware of a CMAKE_BUILD or DUNE_BUILD environment variable.
telling people to generate a config.opts file with the single line CMAKE_FLAGS="-DCMAKE_BUILD_TYPE=Release" works of course but lacks elegance.

DealII for example build by default both a debug and optimized version and they point that out quite prominently.

Another thing mention in the DealII installation is running make in parallel, a feature the use of which could also greatly improve the experience of first time users of dune.

This all assumes, that the user has no knowledge about compiling code using make and cmake at all. Since for both, the procedures are quiet well documented. But, you are right, a well chosen default for the build mode could be helpful, since you always have to specify this. Another option how to set this:

dunecontrol [...] cmake -DCMAKE_BUILD_TYPE=Release [-GNinja]
dunecontrol [...] make [-jN]

I just think, that we cannot document everything for the user that is also common knowledge when compiling code. Students should learn how to use make (and maybe cmake) in the first programming/computer science lectures, for example. Putting more and more into dune-specific scripts like dunecontrol or the opts-file gives the impression that this is the way how you would do this in general. For me, dunecontrol is a script that just executes a command in all/some found dune subfolders in an order given by the resolved dependencies.

It does not hurt to add one line in the config.opts example with

-DCMAKE_CXX_FLAGS=\"-DNDEBUG -O3 -ftree-vetorize\"

since this shows how to actually use your own flags. Experienced users then know what to change and others get a good set of defaults.

The problem is that even somebody familiar with cmake/make might not directly see how to use that knowledge though dunecontrol - since one doesn't directly call cmake or make while building the modules for the first time. You're suggestion above does get around that (even after using ./dunecontrol for ages now I didn't come up with that approach). Perhaps that should be mentioned first in the docu and then the config.opts file as a second step to avoid typing this all out?

My suggestions is to have two dunecontrol, one that simply creates the dependency list and provides the correct cmake call (and maybe install), which then should be a simple script, and another one that has all the fancy things and that should be maintained by someone who actually wants to use all of that.

Students should learn how to use make (and maybe cmake) in the first programming/computer science lectures, for example.

One cannot assume that everybody that uses DUNE has a CS formation. Remember that application installation instructions are almost always documented with dunecontrol. Most of the applications are run by people who are trying to figure out how everything works in a relatively short time so that they can continue with their (probably non-CS/maths related) project.

@santiago.ospina: I absolutely agree. A lot of my students have Python background and do not immediately know cmake and other such things related to the build process. We certainly don't want to limit the use of DUNE to folks that are experienced (Linux) users.

perhaps it is the better option to have a scrip builddune (or whatever) that takes exactly one argument and executes that for each node in the dune module tree. Then Simon's suggestion above would be the default approach. That way it would be clear that we are using cmake/make in the usual way and therefore where to get more documentation from (although I would include the two commands given by Simon in the docu). We don't have to remove dunecontrol of course but we should suggest in the docu to use builddune (or whatever).

@robert.kloefkorn Your point about including something in the documentation so that experienced users know where to change something is very good. Especially, because it is not very clear from the documentation which flags (like CMAKE_FLAGS) exist and what is their effect.

BTW: your way of setting optimization flags might lead to an options clash, in case you combine it with CMAKE_BUILD_TYPE. Then you have some flags set manually and some set implicitly by the build mode. Which one is chosen by the compiler? It would be better to set it on CMAKE_CXX_RELEASE_FLAGS (what has your options as default, I think). But, in principle it shows how to set compiler arguments and that was probably the point.

-DCMAKE_BUILD_TYPE=None

I forgot to mention this.

Not sure, whether splitting the dunecontrol script into two is the solution, but I see the point that there are several commands some have on their wishlist (installing cmake, controlling build modes, maybe downloading modules...). Including everything in one script might lead to a more and more difficult to maintain and to document/describe utility.

Although, it also works when CMAKE_BUILD_TYPE is not specified.

For the current installation instructions on the website, I will make a proposal, including the suggestions from above.

See infrastructure/dune-website!326 (merged) for an initial proposal

@andreas.dedner I added support to easily add a toc, see infrastructure/dune-website!322 (merged)

great, thanks

mentioned in issue #221

I really like the discussion regarding the future direction of dunecontrol. I opened #221 to have all discussions there. Do you mind to move there?

Nobody objects to require CMake 3.13.

@robert.kloefkorn @christi Any objections?

Otherwise I am going to prepare merge requests like !809 (merged) for all core modules and start merge them.

I'm fine with 3.13, but I'm against an immediate switch, until Ubuntu 20.04 LTS is rolled out. So please go ahead and prepare a patch, but wait with merging.

Concerning Ubuntu: I think that at the moment (and until July 2020) users of the 18.04LTS are not encouraged to upgrade to 20.04LTS and have to force the upgrade. That might keep people from doing that. Of course its not too difficult to install by hand or force the upgrade, I guess.

I don't think we will have an other release very soon. And also for master it should not be a problem to delay the switch for two month?!

Require more recent version of CMake

Designs

Child items ...

Activity