VTK tests take a long time in CI

I just noticed that the VTK tests take a long time in the CI system (more than 10 minutes in total for each target). Is there something we can do about that (smaller grids or maybe build those tests with optimization)? Running those tests takes more than third of the overall time required for compiling + running...

One of the problems is that the parallel VTK tests create a second process that eats 100% CPU: the test does most of its work in rank 0 (run some Python stuff), but MPI in rank 1 polls for events.

It would be nice if the idle ranks would actually sleep a bit.

Besides that the Python part seems to be awfully slow :-(

Ah, good to know where the actual problem is - I thought that maybe the grids are just too large. Python code can maybe be optimized...

For reference, the problem is import vtk taking 5s on the CI system (1s on my laptop) per grid:

duneci@c2e728a682f8:~$ time python -c 'import vtk'

real	0m5.516s
user	0m5.292s
sys	0m0.168s

changed milestone to %DUNE 2.6.0

There is https://www.open-mpi.org/faq/?category=running#oversubscribing, but in my experience that does not really help.

Here is an idea, that would help with both the busy rank 1 and the long load time of import vtk. Let the test output a list of files it wrote, then after the test finished pass the list to an external python script. That script does import vtk once and iterates over the list in python.

@dominic The current test only checks whether the files can be loaded by vtk. So while we're at it, I think you have something more fancy in testtools? Would it be worthwile to use that in dune-grid?

Oh, and currently the test reuses the filenames sometimes, we would need to rectify that if we were to implement the above idea.

Hmm, forcing OpenMPI into "degraded" mode might help a bit: Each test uses less ranks than total cores in the test environment, but we might run multiple tests in parallel (in each build and then we also run multiple builds in parallel). OpenMPI doesn't know this and will probably use the "agressive" mode.

Still not optimal, but at least a bit better.

I'll also allow oversubscription at the same time (in case someone sets up a runner with only 1 core).

The relevant changes are here: https://gitlab.math.tu-dresden.de/ansgar/dune-docker/commit/90193c27514c97f1bed0b1f07026cfca74cad60d

I'm guessing that dune_add_test(... MPI_RANKS 2 ...) will run the test simply using mpirun -np 2 test. I would also assume that providing no hostfile is equivalent to a single host localhost. Since providing no slot information in a hostfile is equivalent to one slot per host, I would assume that by default you would only have one slot in total. So OpenMPI should already know that we are oversubscribing, and turn on degraded mode automatically. Of course, man mpirun fails to specify what happens when you do not explicitly specify any host, so my guesses may well be wrong.

@joe: Some versions of OpenMPI didn't allow starting more processes than CPU threads by default (see https://bugs.debian.org/850229) unless one explicitly allowed oversubscription. Though it seems like OpenMPI reverted to the old behavior again (i.e. I can run mpirun -np 32 echo hallo again even when I have less than 32 threads available).

That's why I added rmaps_base_oversubscribe=1 to the Debian packages in February.

Either way, having it set explicitly shouldn't hurt anything.

@joe We have more fancy stuff in testtools, but to be honest I do not think its worth to bring it over to dune-grid, as it is much rather about comparing solutions to reference data than anything else. The only thing that might be interesting: We are not actually using the vtk python package to read the VTK data.

mentioned in commit e12bc772

mentioned in merge request !203 (merged)

closed via commit e12bc772

closed via commit e5f8ea19

mentioned in commit e5f8ea19