Dune PDELab Release 2.7

The existing CI configuration does not reveil issues... I'll update the configuration and open issues for the 2.7 incompatibilities.

changed milestone to %PDELab 2.7.0

removed milestone

changed milestone to %PDELab 2.7.0

@rhess there are some open MRs and some smaller issues, see https://gitlab.dune-project.org/groups/pdelab/-/milestones/2

I would not try to include a new way of calculating the jacobians in 2.7. At the moment I would prefer to:

Just get a 2.7 release done where the CI works.
Get master back to a state where it works with the rest of dune.

Maybe a good way would be:

Branch off releases/2.7 at 33aaa8e2
Have a look if !455 (merged) can be merged into master and backported to 33aaa8e2
Go through warnings in releases/2.7, fix them and merge into master and backport to releases/2.7

If you think another way makes more sense just tell me how I can help :)

@christi Do you think we should follow that approach? I would like to have your opinion before I do anything.

Two comments:

I prefer to fix issues in master and backport them to 2.7
We should go through the list of issues with milestone 2.7.

I think some of these issues are really easy t fix and should thus be in 2.7, while some should not be part of 2.7.

One thing we have to think about is when to branch. We save some work if we first fix and then branch, but this might delay some other MRs...

Why would you branch off at 33aaa8e2? Is there something after 33aaa8e2 which we should keep out?

I would branch off at 33aaa8e2 since the CI is working with 2.7. Then I would fix warnings in master and backport to 2.7. And then I would tackle issues in master and target those for 2.8.

But that is just what I would do and the reason why I asked about your opinion :). We can also do it differently.

Any news here?

As noted in #148 (closed) I just ran some tests with core 2.7 and pdelab and relized that 33aaa8e2 seems to work but 1b77ea94 does not.

(Edit: gcc@8.3.0)

I still need to check if core master + pdelab master works.

Maybe we need to branch off the release branch from the past.

How can 33aaa8e2 work, if it make the CI fail?

On core master + pdelab master the test testordering fails (on donkey with gcc@8.3.0). I will check if this is also the case on my local machine but that takes a bit more time.

I get the following for master (core + pdelab):

fail for testordering on donkey with gcc@8.3.0
pass on donkey with gcc@9.3.0
fail for testordering on laptop with gcc@9.2.1

I probably won't have time over the weekend but I will look at it beginning of next week.

I would highly appreciate it if there was a 2.7 release ASAP. I do not care for any additional features---the status quo is fine with me (as long as the CI passes).

@rhess @santiago.ospina: as discussed with @oliver.sander today on the phone, I suggest to have a video-conference regarding a 2.7 release on monday or tuesday.

Topic we need to discuss:

which issues are show-stoppers?
a time line
when to create a branch?
branch-point for 2.7

Feel free to add more topics and inform other people which might be interested in joining.

I suggest now a meeting on monday 16:00 CEST (UTC+2). We could use https://meet.jit.si/ to meet.

Please reply to this comment and leave me a note it you want to join. I can create a session and send the details around.

For me monday 16:00 CEST is good :)

As a mere user that needs a 2.7 release I am not sure whether my presence at the meeting is helpful. But if you think it is I'll be happy to join. 16:00 CEST is okay for me.

@oliver.sander I don't think it is necessary that you join, except if you have particular remarks you'd like to add to the discussion. @santiago.ospina would you be able to join at 16:00?

Thanks. Indeed I don't think I'll be able to join, for private reasons.

Argh! For some reason my email-response was not added here.

We can meet at https://conf.wwu.de/ room pdelab

So the plan for now is the following:

Bring PDELab master in a state that works with master and 2.7 of the core modules.
Create a release branch
Tackle milestones from https://gitlab.dune-project.org/groups/pdelab/-/milestones/2

I will try current master with gcc7 and clang 5 and report back tomorrow.

Unfortunately I cannot build with clang5 yet since spack behaves weired. I can make test runs with gcc@7.1.0.

For core@2.7 there is still the tree path issue #148 (closed)
For core@master we also have various errors

I guess the first step is fixing those errors. I will spend some time on this this evening.

I'm not sure if the plan above still makes sense. If we try to make pdelab master work with core master and 2.7 we run into the problem of Std::make_index_sequence being replaced by std::make_index_sequence. It would be easier to just make pdelab compatible with core@2.7, branch off and then make it compatible with master.

Opinions?

I'm not sure if the plan above still makes sense. If we try to make pdelab master work with core master and 2.7 we run into the problem of Std::make_index_sequence being replaced by std::make_index_sequence. It would be easier to just make pdelab compatible with core@2.7, branch off and then make it compatible with master.

There are currently 4 spots in pdelab using Std::make_index_sequence. I think these can easily be managed. And currently we still have std::make_index_sequence imported into the namespace std in dune-common.

assigned to @rhess

@rhess I assigned to you as I hope that you will keep an eye on this...

changed the description

I updated the tags of the 2.7 issues and added a checklist to this page.

Ups, sorry! I didn't see this on time. Will fix my email notifications in order to be more responsive to direct calls to me.

@rhess I would like to deprecate the old DiscreteGridFunction implementations. We have an implementation of the dune-functions style interface and having two interfaces around will lead to chaos on the long run.

do you see a major obstacle?
currently we only have the gradient type derivative. Is this sufficient?
are there features we need to add? E.g. do we need curl(fnkt)?
how should we handle pull-backs?

I always thought that the new one is not yet able to handle vector and power function spaces? That would be something I'm concerned about.

Regarding pull-backs: I guess the question is if the pullback is done automatically or if it is done outside by the user? From a performance point of view it makes sense if it is possible to apply it manually.

I don't have strong opinions on the derivative stuff but I maybe others will have to say something. @dominic ?

As a user of DiscreteGridFunction in DORiE and dune-copasi I wouldn't be very happy if the transition to dune-functions requires me to know the internals of the two versions in order to upgrade my code.

As the Hybrid Tree Path issue is fixed (thanks @christi !) I now ran our tests with different compilers and varying results. I ran pdelab@master with core@master and with core@2.7. Below is a summary:

	core@master	core@2.7
gcc@7.1.0	fail	fail
gcc@8.3.1	fail	success
gcc@9.3.0	fail	success
clang@8.0.0	fail	fail
clang@9.0.1	fail	fail

For the core@master it fails this way:

gcc@7.1.0: Building the tests
gcc@8.3.1: Execution of testordering (see !492 (merged))
gcc@9.3.0: Execution of testordering
clang@8.0.0: Execution of testordering and testpk2dinterpolation
clang@9.0.1: Execution of testordering and testpk2dinterpolation

For core@2.7 I get the following problems:

gcc@7.1.0: Execution of several nonoverlapping tests (testnonoverlappingsinglephase... and testnonoverlapping-mpi-2)
clang@8.0.0: Execution of testpk2dinterpolation
clang@9.0.1: Execution of testpk2dinterpolation

On my local machine I could confirm the errors for gcc9 and clang9 for master and 2.7. If someone has a working gcc@7 it would be interesting to see if it behaves similar.

I will do more tests tomorrow and see where that leads...

I will try gcc 7+8.

Update

I fixed the testordering and testpk2dinterpolation tests and Dominic added CMakeGuards and tested with gcc@7.5.0 and clang@6. Now the tests should pass for core@master and core@2.7 for gcc>=7.5.0 and clang>=6.?.?.

Regarding gcc@7.1.0: This could also be an issue of the gcc7 compiler on our compute server.

Warnings

Regarding warnings there seem to be 3 classes:

Warnings regarding Hybrid Tree Path
Warnings regarding Dune::Function deprecation. This probably needs some fixes in dune-functions first (see staging/dune-functions#57 (closed)) and afterwards some changes in pdelab.
Other warnings. I can't tell how many there are since the other two cases produce a huge amount of output. I don't think there are many.

When to branch off

@christi: What is your opinion on branching off? If we first want to tick off milestones (https://gitlab.dune-project.org/groups/pdelab/-/milestones/2) I propose:

A start for #155 (closed) is done in !493 (merged).
Do things regarding pdelab-tutorials after branching off.
Postpone #106, #66 (closed), #87, #86, #45, #66 (closed)
Merge !490 (merged) to close #138 (closed)

This leaves:

#150 (closed): Status unclear?
#102: Status unclear, probably complicated and annoying?

New merge requests

I don't care if the matrix-free stuff is part of the release or not. There is a new merge request by @linus.seelinger. Maybe this could be included? We should discuss over there: !481

Packaging tar

In the description above creating tars in mentioned. In my opinion we don't need to do that (you can also get tars through gitlab). Instead I propose rewriting https://www.dune-project.org/modules/dune-pdelab/. I would volunteer to do that.

I think we can branch off. My main concern initially was, that we can't easily backport from master to 2.7.

Still things would be easier, if we keep both in sync.

Regarding your list:

A start for #155 (closed) is done in !493 (merged).

we could merge and then tackle the old Newton, butI would actually prefer doing this right away.

Do things regarding pdelab-tutorials after branching off.

agree

Postpone #106, #66 (closed), #87, #86, #45

we can still backport these, if we fix them in a timely manner

Merge !490 (merged) to close #138 (closed)

this can be done right away, right?!

#150 (closed): Status unclear?

I suggested a fix...

#102: Status unclear, probably complicated and annoying?

imho this must be fixed in the one or the other way

I ran all the tests with gcc 7,8,9 and clang 6 and master seems to be in a pretty good state now. I also started to work on removing warnings, the only big one left in this regard is the TreePath one that I did not look into:

warning: ‘using TreePath = class Dune::TypeTree::HybridTreePath<Dune::index_constant<i>...>’ is deprecated: use StaticTreePath, this type will be removed after DUNE 2.7 [-Wdeprecated-declarations]

What is the fix for this one? I also agree with everything @rhess and @christi said above about how to proceed except that I still have no idea about how to fix #102.

Where did you get the warning? I thought I removed any direct use of TypeTree::TreePath ...

The usage seems to be typetree internal (e.g. in childextraction.hh):

In file included from /home/domse/structures/dune-typetree/dune/typetree/powernode.hh:17,
                 from /home/domse/structures/dune-typetree/dune/typetree/typetree.hh:10,
                 from /home/domse/structures/dune-pdelab/dune/pdelab/common/function.hh:17,
                 from /home/domse/structures/dune-pdelab/dune/pdelab/boilerplate/pdelab.hh:59,
                 from /home/domse/structures/dune-pdelab/dune/pdelab/test/testpoisson-periodic.hh:10,
                 from /home/domse/structures/dune-pdelab/dune/pdelab/test/testpoisson-periodic-2d.cc:7:
/home/domse/structures/dune-typetree/dune/typetree/childextraction.hh:168:60: warning: ‘using TreePath = class Dune::TypeTree::HybridTreePath<Dune::index_constant<i>...>’ is deprecated: use StaticTreePath, this type will be removed after DUNE 2.7 [-Wdeprecated-declarations]
  168 |     decltype(auto) child (Node&& node, TreePath<Indices...>)
      |                                                            ^
In file included from /home/domse/structures/dune-typetree/dune/typetree/childextraction.hh:16,
                 from /home/domse/structures/dune-typetree/dune/typetree/powernode.hh:17,
                 from /home/domse/structures/dune-typetree/dune/typetree/typetree.hh:10,
                 from /home/domse/structures/dune-pdelab/dune/pdelab/common/function.hh:17,
                 from /home/domse/structures/dune-pdelab/dune/pdelab/boilerplate/pdelab.hh:59,
                 from /home/domse/structures/dune-pdelab/dune/pdelab/test/testpoisson-periodic.hh:10,
                 from /home/domse/structures/dune-pdelab/dune/pdelab/test/testpoisson-periodic-2d.cc:7:
/home/domse/structures/dune-typetree/dune/typetree/treepath.hh:433:11: note: declared here
  433 |     using TreePath [[deprecated("use StaticTreePath, this type will be removed after DUNE 2.7")]] = HybridTreePath<Dune::index_constant<i>...>;
      |           ^~~~~~~~
In file included from /home/domse/structures/dune-typetree/dune/typetree/powernode.hh:17,
                 from /home/domse/structures/dune-typetree/dune/typetree/typetree.hh:10,
                 from /home/domse/structures/dune-pdelab/dune/pdelab/common/function.hh:17,
                 from /home/domse/structures/dune-pdelab/dune/pdelab/boilerplate/pdelab.hh:59,
                 from /home/domse/structures/dune-pdelab/dune/pdelab/test/testpoisson-periodic.hh:10,
                 from /home/domse/structures/dune-pdelab/dune/pdelab/test/testpoisson-periodic-2d.cc:7:
/home/domse/structures/dune-typetree/dune/typetree/childextraction.hh:178:57: warning: ‘using TreePath = class Dune::TypeTree::HybridTreePath<Dune::index_constant<i>...>’ is deprecated: use StaticTreePath, this type will be removed after DUNE 2.7 [-Wdeprecated-declarations]
  178 |     auto childStorage (Node&& node, TreePath<Indices...>)
      |                                                         ^
In file included from /home/domse/structures/dune-typetree/dune/typetree/childextraction.hh:16,
                 from /home/domse/structures/dune-typetree/dune/typetree/powernode.hh:17,
                 from /home/domse/structures/dune-typetree/dune/typetree/typetree.hh:10,
                 from /home/domse/structures/dune-pdelab/dune/pdelab/common/function.hh:17,
                 from /home/domse/structures/dune-pdelab/dune/pdelab/boilerplate/pdelab.hh:59,
                 from /home/domse/structures/dune-pdelab/dune/pdelab/test/testpoisson-periodic.hh:10,
                 from /home/domse/structures/dune-pdelab/dune/pdelab/test/testpoisson-periodic-2d.cc:7:
/home/domse/structures/dune-typetree/dune/typetree/treepath.hh:433:11: note: declared here
  433 |     using TreePath [[deprecated("use StaticTreePath, this type will be removed after DUNE 2.7")]] = HybridTreePath<Dune::index_constant<i>...>;
      |           ^~~~~~~~
In file included from /home/domse/structures/dune-typetree/dune/typetree/powernode.hh:18,
                 from /home/domse/structures/dune-typetree/dune/typetree/typetree.hh:10,
                 from /home/domse/structures/dune-pdelab/dune/pdelab/common/function.hh:17,
                 from /home/domse/structures/dune-pdelab/dune/pdelab/boilerplate/pdelab.hh:59,
                 from /home/domse/structures/dune-pdelab/dune/pdelab/test/testpoisson-periodic.hh:10,
                 from /home/domse/structures/dune-pdelab/dune/pdelab/test/testpoisson-periodic-2d.cc:7:
/home/domse/structures/dune-typetree/dune/typetree/typetraits.hh:170:17: warning: ‘using TreePath = class Dune::TypeTree::HybridTreePath<Dune::index_constant<i>...>’ is deprecated: use StaticTreePath, this type will be removed after DUNE 2.7 [-Wdeprecated-declarations]
  170 |         -> std::true_type
      |                 ^~~~~~~~~
In file included from /home/domse/structures/dune-typetree/dune/typetree/childextraction.hh:16,
                 from /home/domse/structures/dune-typetree/dune/typetree/powernode.hh:17,
                 from /home/domse/structures/dune-typetree/dune/typetree/typetree.hh:10,
                 from /home/domse/structures/dune-pdelab/dune/pdelab/common/function.hh:17,
                 from /home/domse/structures/dune-pdelab/dune/pdelab/boilerplate/pdelab.hh:59,
                 from /home/domse/structures/dune-pdelab/dune/pdelab/test/testpoisson-periodic.hh:10,
                 from /home/domse/structures/dune-pdelab/dune/pdelab/test/testpoisson-periodic-2d.cc:7:
/home/domse/structures/dune-typetree/dune/typetree/treepath.hh:433:11: note: declared here
  433 |     using TreePath [[deprecated("use StaticTreePath, this type will be removed after DUNE 2.7")]] = HybridTreePath<Dune::index_constant<i>...>;

Thanks! This should be fixed with the latest typetree merges...

Thanks @christi. There are some few warnings left after !497 (merged) which I will tackle in one combined merge request.

With !497 (merged) and !499 (merged) I am down to 0 warnings and I am in favor of branching off after merging these. In !497 (merged) there is a bit of discussion left @christi

We have to find a fix for #102, but we should still be able to backport...

marked the checklist item Bring dune-pdelab master in a state that works with master and 2.7 of the core modules. as completed

Great! I am currently testing again locally against the releases of the core modules and will create the release branch as soon as that succeeds.

Branch created!

marked the checklist item Create a release branch as completed

@oliver.sander we now have a 2.7 release branch...

Thank you guys! Ansgar has already uploaded several 2.7 packages to Debian, he may be interested in this one, too.

I tried last week and noticed that dune-typetree needs a 2.7.1 release.

@ansgar might want to wait a few more days, we have some other changes in the pipeline that should be backported...

Agree with @christi. Also some additional testing should be conducted.

I suggest to meet at 16:00 at https://conf.wwu.de room pdelab

closed

Dune PDELab Release 2.7

Designs

Child items 0

Activity

Update

Warnings

When to branch off

New merge requests

Packaging tar