Performance testing - Evaluate the deal.ii approach
I talked to Timo Heister from the deal.ii team at SIAM CSE17 and he developed a new tackle on performance testing. We should have a look and decide whether we want to follow the same route.
Here is the summary:
- Instead of measuring times, which is a noisy, clunky procedure, they start measuring instruction counts.
- Instruction counts are deterministic, so they can be compared with very low tolerance values
-
valgrind --tool=callgrind
and some additional gymnastics do the job - The code has to be instrumented with macros starting and stopping the measurements (the counting only really makes sense in innermost loops, like assembly, iterative solvers etc.)
- deal.ii uses this on a monthly basis with
git-bisect
(which is a nice alternative to CI) - Timo has uploaded his code to github