Improve the CI systems tests

Summary

Currently all commits in dune-common trigger a pipeline in the https://gitlab.dune-project.org/infrastructure/dune-nightly-test repository to test downstream modules. It runs the pipeline on the branch core that contains only the core modules in a few compiler configurations. This MR proposes to 1. provide a trigger for the full-system test as well and 2. make the downstream pipelines a job that can be manually started.

Discussion

• Is there a pipeline that should run on all commits? --> no
• Do we want to require that a specific pipeline succeeds before we can merge into master? --> yes: core-modules
• Do we want to run both, the core-modules and the full-system pipeline? --> (yes): allow both to be run manually, system-test has "allow_failure"

mentioned in merge request !1201 (closed)

What is the reason to distinguish between merge_request_event and push? What we do in dumux is always having all pipelines on manual but configure GitLab so noone can merge without the pipeline passing. That way, eventually you have to manually run the pipeline (otherwise you cannot merge) but it's never triggered automatically avoiding overhead.

added 1 commit

• d5c1f933 - Add playbutton for core and full-system tests

Compare with previous version

marked this merge request as ready

mentioned in merge request !1345 (merged)

Let's think again:

• We could have two types of pipelines: branch and merge_rquest. The first one runs on the current branch, the other on the target branch if it would have been merged already.
• For me the merge_request pipeline is more important to have a system test passed. The branch-pipeline might even sometimes not be possible to be used with the master of other modules - think about a change in the buildsystem.
• manually triggering the system or core downstream pipeline is better than automatically running everything. (no waste of computing time, the branch must compile first and all changes must be incorporated before we even try to run the full system test)
• What do we want to enforce? That all core modules compile fine with the changes in a branch, or that the whole system passes? The second implies the first, so we don't need to have both as a requirement.
• How strickt do we want to be with the downstream pipelines? Only require that the branch pipeline passes? Also require that the core modules pass, or that the full system passes? Forbid to merge with failed pipelines, or still allow a forced merge?

Currently, we have to trigger system tests and external python test on gihub manually. This is a situation I definitely want to change. I am not sure whether we can fix the situation with pipelines on gihub, but at least we should start with the system test here in our gitlab.

We could even add the core-module-test and full-system-test into the ci-config include files, such that it is part of all core modules. Maybe we add a second central config file with these two new jobs.

Another consideration is that currently, push events without merge requests are being run. This is a consequence of ci-config!19 (diffs) where I had to make the switch between merge requests and push events. This is also a waste of resources because once the merge request is open, another pipeline is activated. I think we should simply not run pipelines on commits other than the protected ones (e.g. master, releases/*, omp/*) without merge requests because our policy is that everything has to be funneled to the protected branches through merge requests.

mentioned in merge request !1358 (merged)

changed title from Make systemtest a manual startable job to Improve the CI systems tests

changed the description

added 1 commit

• bcbd8dae - Add merge-request pipeline

Compare with previous version

added 319 commits

• bcbd8dae...7c4b51c2 - 315 commits from branch master
• a5361224 - Make systemtest a manual startable job
• f2413bc9 - Add playbutton for core and full-system tests
• 0482f3ef - Add merge-request pipeline
• 0202b233 - Add push and merge-request pipelines

Compare with previous version

added 1 commit

• 400ce6e5 - Make systemtest a manual startable job

Compare with previous version

@core Very often we have the situation that we want to run system tests to check that changes to dune-common do not break downstream modules. Currently we only run core module tests automatically. These tests run on all commits. This is often not needed and a waste of resources. But we want to have the guarantee that on merge into master the downstream modules are not broken. The solution proposed here is to make the systems tests a job that can be manually started. In the pipeline overview or summary in the MR you will see a small gear symbol next to the green light bulbs. This gear symbol allows to start a downstream pipeline directly from the MR.

Two downstream pipelines to start manually are provided:

1. A core-module pipeline,
2. A full-system pipeline.

The core-modules downstream pipeline must pass in order to allow merge. This is now enforced strictly in the repository setting (Merge checks: Pipelines must succeed). Thus, after finishing the MR, a downstream pipeline for the core modules must be manually triggered and then succeed before the merge-button will be available.

Full-system tests are also provided as manual pipeline. Currently we do not enforce that this downstream pipeline must succeed. We have lots of modules in this system test and some are not updated frequently. So, I propose to 1. allow running the system test directly from the MR, but 2. allow failure of this pipeline. Still we will get an orange light bulb to investigate possible failures. It can be decided later whether we want to implement a different strategy.

As discussed in the comments above, we can add this manual core/system-test also to other (core) modules, e.g., in the job-templates.

Good idea. Thank you for working on this proposal!

Can we adjust the images? We have three tests with Debian 11 and four tests with Ubuntu 20.04. We are not testing recent compilers, CMake, and dependencies. Can be please change one image to Debian 12 and another to Ubuntu 24.04 or 23.10?

Sure, the system test can be improved. In https://gitlab.dune-project.org/infrastructure/dune-nightly-test we have a python script that generates the .gitlab-ci.yml file and there one can easily update compiler/toolchains and add modules/configurations.

Great suggestion. I'm all for it.

With this MR we should close !1201 (closed) and !1202 (closed)

approved this merge request

Any reasons not merge this? If you object or need more time, please comment. Otherwise I am going to merge.

merged

mentioned in commit 6d6e5fa2