test-parallel-ug (from dune-grid) fails
The test-parallel-ug
test from dune-grid
fails.
Some observations:
- It crashes on 3D cube grids. 2D cube grids and 2D or 3D simplex grids are okay.
- It only crashes with adaptive refinement. Global refinement is okay.
- It only crashes with >= 3 processes.
- It crashes as a side vector is attached to the fifth side of a tetrahedron (which leads to invalid memory accesses a bit later).
What is special about the error case compared to the others?
- It uses red/green refinement (cube grid with adaptive refinement in 3D).
- There exists a processor boundary with green elements on both sides. (But these also exist with 2 processes.)
When switching the role of x- and y-coordinates (refine for y < 0.5 instead of x < 0.5), the test suddenly passes with 4 processes, but still fails with 3 or 8 processes. So the direction seems to be important. (See images at the end.)
In conclusion I suspect there is some issue with green/green refinement over process boundaries: depending on the direction (or order of processes?), sometimes side vectors are attached to the wrong side (or wrong element?).
If that's not the issue, maybe load balancing causes the problem? After all the load balancing test doesn't pass either.
Images
These show two runs of test-parallel-ug
with 4 processes.
In the first run, elements with x < 0.5
are refined and the program eventually crashes:
In the second run, elements with y < 0.5
are refined and the test passes (but only with 4 processes, not with 3 or 8):
The colors for each rank are the same for both runs. The viewing direction differs (x and y axes are switched).