Edges in parallel UG are broken
When compiled in parallel mode the edges in a parallel dune-uggrid are broken. More specifically: In a 3d grid obtained from successive local refinement of a simplex the leaf grid contains edges that have a non-trivial intersection. E.g. they intersect in a point that is neither a vertex of one or the other edge. When compiled in sequential mode this does not happen. Moreover, the number of total edges is larger in parallel mode while the number of elements, faces, and vertices is the same. One can also observe, that the centers of the elements are different.
The attached example demonstrates the issue: uggrid-failure.cc . It works nicely when dune is set to sequential mode and the edge count is different.
I was unsure if this should be files here or for dune-uggrid. Since I have a vague memory that UG does not provide edges by itself and that they are implemented in the dune bindings I picked dune-grid.
Activity
Can you install a sequential and a parallel version, and step through the two side-by-side with the debugger?
Both versions use the same code paths, only the parallel version has a few extra bits guarded with #ifdef ModelP. It should be relatively easy to find the spot where the sequential and parallel code deviates.
Milestone changed to %DUNE 2.5.0
Confirmed. On a sequential installation I get
Number of codim-0 entities: 518 Number of codim-1 entities: 1160 Number of codim-2 entities: 804 Number of codim-3 entities: 163while with MPI it says
Number of codim-0 entities: 518 Number of codim-1 entities: 1160 Number of codim-2 entities: 805 Number of codim-3 entities: 163That one will be fun...
Edited by Oliver SanderThere is something really fishy going on here. The resulting grid contains vertices that are neither vertices nor edge midpoints of their father elements. That should not happen on a simplicial grid, right?
And what is worse: this even happens without MPI.
New shorter test case uggrid-failure.cc attached.
Edited by Oliver Sander
I feel really uncomfortable about staging/dune-uggrid@5f06ebe2 by now (I somehow cannot accept that the code would do anything remotely reasonable both before and after the change so while a bug report would have been a good idea my patch was overeager I'm afraid) and even though I'm pretty sure I was seeing this error already before the commit in question (unfortunately it seems I made the commit and the bug report the same day), I feel that I need to ask you to please make sure that the weird error you're seeing is in no way related to my commit (e.g. by reverting the commit in question at least locally and testing again). Sorry and thanks.
-
This is what I get when I add uggrid-failure.cc as a test to dune-grid and run uggrid-failure:
DimX=1, DimY=1, DimZ=1 DimX=1, DimY=1, DimZ=1 Adaptive refinement ended Number of codim-0 entities: 518 Number of codim-1 entities: 1160 Number of codim-2 entities: 805 Number of codim-3 entities: 163 level 1 level 2 level 3 Error: element corner is not midpoint or corner of the father Corner: 0.8125 0.0625 0.0625 Corners of the father: 0.75 0 0 0.75 0.25 0 0.75 0 0.25 1 0 0 [mojito:06135] *** Process received signal *** [mojito:06135] Signal: Aborted (6) [mojito:06135] Signal code: (-6) [mojito:06135] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0) [0x7fe126b198d0] [mojito:06135] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37) [0x7fe126794067] [mojito:06135] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7fe126795448] [mojito:06135] [ 3] ./dune/grid/test/uggrid-failure(main+0xd67) [0x456537] [mojito:06135] [ 4] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7fe126780b45] [mojito:06135] [ 5] ./dune/grid/test/uggrid-failure() [0x4556f9] [mojito:06135] *** End of error message ***Edited by Elias Pipping I can't find it in the code right now, but maybe UG does deliberately introduce element center vertices for tetrahedra under certain circumstances. I have a hard time believing that I really did find a bug in UG's refinement implementation for simplicial grids. I think I'll go back to Carsten's original testcase.
I did some more digging. In the file refine.cc there is the method RefineElementGreen which computes green closures. On line 4968 it says
#ifdef ModelP unsigned int maxid = 0; #else INT maxid = -1; #endif [snip] /* if (node0 == maxedge && ((SIDEPATTERN(theElement)&(1<<i)) == 0)) */ if (_ID_(theSideNodes[node0]) == maxid) {Indeed, at the offending element, the sequential uggrid takes the 'if', but the parallel one takes the 'else', and that apparently leads to the observed symptoms.
Unfortunately, I have currently no idea what the code in [snip] does. It appears to be there for a reason. Simply removing the #ifdef ModelP doesn't help: it makes the code run into an assertion right away. And even if it did fix the issue I would not want to remove it without understanding what it does, for fear of regressions in parallel code.
Most likely I will not be able to spend more time on this in the nearer future. Feel free to pick it up from here if you are motivated.
changed milestone to %DUNE 2.6.0
mentioned in commit staging/dune-uggrid@5b81870c
mentioned in merge request staging/dune-uggrid!75 (merged)
I asked @peter about this, and he suggested that we should really run UGGrid with the TET_RULESET macro set. Doing so does indeed fix the problem. staging/dune-uggrid!75 (merged) introduces a CMake flag that lets the user set TET_RULESET. It seems to work okay, but I'd rather not enable it unconditionally without further tests. Also, it expects to read a list of refinement rules from a file, which is a bit inconvenient.
mentioned in issue infrastructure/releases#1
added bug label
@oliver.sander: Fine with me, thanks!
changed milestone to %DUNE 2.7.0
-
@oliver.sander: Marked for 2.7 now. Please don't forget to add the changelog entry.
Fixed in staging/dune-uggrid!134 (merged)
