Where do reference solution and data come from?
Having reference data is necessary for comparing results to results of previous runs. It is not obvious, however, how an automated system should handle reference data. Some possible approaches are:
- It is in the source tree, under Version Control
- PROS: easy to implement, ensures long term tracking (no system artifacts may alter the reference solution)
- CONS: possible a (too) large amount of data (vtus), commiting new reference solutions after a test is changed
- The reference data is generated by the build system
- PROS: no data in VCS
- CONS: After a fresh checkout, you have ot simply assume, that your tests are doing the right thing while generating reference solutions, the CMake implementation of something like "make reference" is rather big and ugly.
- Doing a before/after comparison
- PROS: no data in VCS, reference solutions may change without maintenance of the solution
- CONS: doing a checkout for generating the reference solution is hard in a local environment, Slow regression (though those are arguably not a problem with reference solutions.) In a server environment, the solution can be stored on the master server.
- Combining 2 and 3
- Leads to an approach of the form: If I cant find a solution, rewind the clock to get one. This negates the first CON of 2.
For simplicity, we use 1 for now. This should still be discussed.