[discussion] drop or simplify pip & venv magic during configure

As we have seen in !1235 (merged), the automagic handling of python dependencies leads to a couple of strange problems, e.g. packages installed automatically by cmake are not always found.

I would like to start a discussion on how we can simply this whole cmake-venv-pip integration. My provocative statement is that it leads to more problems than it solves and that a clear error message about missing dependencies would be far more helpful.

Opinions?

added buildsystem discussion python labels

@christi I'm not quite sure what your suggestion is and where you would like more clear error messages?

A suggestion on how to simply it would be great. But for me the main problem has always been that we could not decide on a common workflow. Some need the internal venv, some use an external one, some refuse to use any venv at all. Adding to that the issue of having pypi packages and cmake build folders and you end up with with at least half a dozen different way the bindings could be used. That causes the problem not having some dependencies. As long as we can't decide on a common way of using Python we will not be able to simplify.

Concerning the mpi4py problem, that came up because mpi4py was now used while compiling a library outside of the dune-py JIT context and that could happen without an active venv. A way around that is of course to JIT compile the communicator on demand then that problem goes away. The clear lesson is that venv dependent code should not be used in files like `_common.cc but that is not really an issue requiring to refactor the Python cmake infrastructure yet again.

Exactly these different ways how you could use DUNE and all being subtle different leads to a lot of confusion. I had now a couple of students working with DUNE via python, but it never worked as simple as hoped for. I'm sure that it works better, if the advisor is a python guru, but this is not what we can expect from our users.

The usual way of enabling different use cases is with a modular design. This is what Dune is supposed to be good at. But it only seems to hold for the C++ part of things. Particularly not for CMake (improvements on the way) and the Python setup.

I'm going to make the unpopular claim that IMHO enabling the Python bindings per default was a wrong decision:

(Simply changing the default of course doesn't fix all problems with the setup. But I think the fundamental problems start there.)

Currently, the way things work are essentially that the default is a lot of magic, pip, venv that can't be expected from a (mainly) C++ code. Different workflows are then enabled by providing switches to tune the magic or switch off parts of it. These workflows are also not well documented. But this is not the main issue. The main issue is, why is all of this happening if I haven't asked for it? (including many problems that can come up during the setup if things go wrong, which is basically every time a new person tries things out).

The big advantage of actively opting into features is that it's much easier to come up with useful error messages and explain what went wrong because the user explicitly gave instructions. For example, one may require mpi4y when the user asked for Python bindings that run in parallel. If they explicitly ask for it, you can simply say: You asked for it and this comes with the requirement that mpi4py needs to be found but it was not found. If it's the other way around (i.e. we enable by default) then we have to make magic decisions. E.g. what do we do if mpi4py is not found but MPI is there, error? silent ignore? warning? I would argue the best message depends then on the user and for which case they want to use the program. We can't know that in advance. This means only a compromise decision can be made in advance instead of what is the most informative message for the specific user.

My second major point is that if a setup is complex and developers still want to contribute, at least the CI system should give people the confidence to change stuff without the fear of breaking things:

This is independent of Python but the Python situation makes this obvious. We all rely on the CI to tell us if things go wrong because no one has the capacity and knowledge to know about every single corner case. In particular new developers (which we might want to attract :) and not repel). I see two problems with the CI: (1) The first obvious thing with the Python bindings is that there is a separate module on GitHub that tests essential components but that is not integrated at all into the GitLab CI system. This is simply not a practical developer environment. One CI system should handle all cases and a developer should be able to start things in one place. (2) The second problem is that the CI system in Dune seems over-engineered to me. It is extremely hard to find the script that is executed and what is actually done to test. Testing should be something everyone can repeat on their own machine, so preferably the setup should also be simple enough to write it in some lines of code. The CI system is spread over multiple repositories with many magic switches that only some inner circle understands. Developers are also users.

Related issues

There are many other open issues regarding the Python setup: #138 (closed), #194 (closed), #226, #232 (closed), #239 (closed), #247, #248 (closed), #273, #278 (closed), #279, #280 (closed), #281, #289, #294, #300, #311, #313, #325 (closed), #327 (closed), #328, #329. Easy to lose overview.

I'm going to make the (possibly un)popular suggestion that we should move everything back into dune-python

That is then a clear opt-in as requested and I can just get on again with doing things as I want to do them without having to try to satisfy everyone else.

I'm opting out of this discussion btw. - I've already spent more time then I can afford to get things this far and I am not going to get involved in another rewrite! I'll just open up issues and revert things if my current workflow stops working.

I'm also for adopting a code of conduct that all developers agree on. The goal would be to accept that opinions can be voiced without others making fun of them.

I was not making fun of anyone or anything just stating my favorite solution to the problem. Integrating this into dune-common was a clear mistake since we can not get it to work, It did not work with opt-out either so let's move it out of dune-common again into dune-pthon and then the opt-in is clear. I also just stated that my capacity for working on this is exhausted but I want my current workflow to continue to work. Don't know where there was any making fun there. But feel free to setup a guideline - I'm sure there is some available and then point out were I got it wrong and I'll apologies.

I don’t think that ripping python support out again is a good option.

One part of the reason that we are currently facing different issues with the python bindings is that they are a success! People want to use them and try to implement methods with increasing complexity or at least methods requiring features not yet supported.

As @tkoch listed, I also see a couple of problems in the current cmake/python integration. This doesn’t question the python interfaces at all and I’m sorry if I made a wrong impression here.

Still I think there must the possibility to critically review the current situation and discuss how to improve things. And I also think this is something we as active developers must be willing to do (to a reasonable extent). If I as an experienced dune dev don’t understand how I can contribute without breaking everything, how can we then attract new developers? Again this is not intended as a personal criticism, all I want is to improve the status quo (and hopefully without too much disturbance).

@andreas.dedner is abosultely right.

I looked through the issues and everything up to #279 is stale and I closed most of those issues. Would have been nice if that would have been done when the list was created!!! It is also noticebale that all these issues contain comments and fixes from @andreas.dedner, so we should all be thankfull for the enourmous effort.

So it remains: #279, #280 (closed), #281, #289, #294, #300, #311, #313, #325 (closed), #327 (closed), #328, #329, out of which only a few are very critical and some are debatable.

It might be a good idea to have a two day DUNE-dev meeting to discuss the cmake+Python BS (build system) and fix the issues right there and then.

@tkoch: The github CI was created because it allows testing on MacOS, which we do not provide currently. It would be appreciated if you can help to integrate some of the tests into the https://gitlab.dune-project.org/infrastructure/dune-nightly-test system.

And yes, I also get confused with which scripts are run and why tests are failing. And when tests fail, then it's very diffcult to figure out why and where and on which strange debian/clang combination.

I think this discussion is too important to leave many people out. Pinging @core.

To me, the Python bindings look like a great success and people are willing to use it. Many thanks to Andreas for his continued effort to fix other people's issues and helping with constructive input in various discussion. Similar to Timo, Carsten and others, that picked up the good work and making it even better (the power of collaboration)!

Nevertheless, as a non-Python user it looks like a big black box and I don't understand all the magic done by our build system.
For me, the Python bindings should be reduced to direct dependencies on the user's machine (Python, py bind, a couple of Python libraries). If one of the required Python dependencies is not fulfilled, the Python binding is disabled. That is a reproducible way, for users, CI/CD pipelines, and distribution package builders. For me, everything else like venv would be sugar on top and might become opt-in, non-backwards-compatible, with reduced testing, and additional limitations, I cannot even formulate. This should reduce the burden on keeping this magic working for too exotic use cases. And it should help for new users to guide them on a recommended, hopefully safe path. I might be wrong, and others have a broader picture compared to me.

Today there are several options already:

1. Default: Python enabled and a dune-env is configured inside dune-common build dir to enable pip based package installation. Keep in mind, not all Python packages currently used might be available as system packages, e.g. portalocker and others.

2. Python no: which can be achieved with -DDUNE_ENABLE_PYTHONBINDINGS=OFF when calling dunecontrol. Then all Python support should be turned off and heopfully this works as expected.

3. Python yes, dune-env no: This is achieved with -DDUNE_PYTHON_USE_VENV=OFF and should pretty much behave like @tkoch wants to have it. It certainly does what I want. When using this option Python packages have to be provided by the user, which is some cases requires manual installation.

So which combination is missing? It can't be that difficult to one flag to config.opts to select one of the avove options. I suggest to not support more options, because it just makes it ever more difficult to test all combinations. As far as I remember the dune-env business was added to support dune-testtools and the Python bindings in one setting.

So in order to drop pip during configure option 3) should be selected. Once this is working flawlessly the default can be switched from ON to OFF.

1a: no venv is active during run of dunecontrol: internal venv setup and packages are installed there

1b: external vemv is active during run of dunecontrol: packages are installed there no internal venv used

Hi all, sorry for my reticent reply! I just started a new job and will have limited capacities for DUNE in the future. However, I understand the desire to simplify things here and am willing to contribute to improve this!

My thoughts and ideas on how to simplify things:

1. Spell out setup.py & pyproject.toml (instead of calling metaData function)

   • Makes more verbose what’s going on, including explicit versioning
   • Both files are part of git for an obvious reason
   • pip install . will become more reliable and explainable
   • Downside: One has to update these files manually for every module and version (but we can keep the packaging script to autogenerate/update these files)

2. Make the automatic venv setup optional per default (or remove it / move it to another module) [sorry Andreas, unpopular opinion, as especially you put so much work into this setup...]

   • Venv setup originated by Dominic, mainly because of embedded python. Is anybody actually using this feature? If not, we shouldn't put effort in supporting it because of this reason.
   • I like the simplicity of the venv setup if everything works as expected, but I also observed situations where it was very hard to find the reason for a problem if something went wrong.
   • Alternative: As proposed, user sets up his python environment manually (or let it do by an optional tool)
     • Workflow would be:
       • Clone dune modules, configure, build (including python libs per default, if possible)
       • Call pip install in every module, which installs also python dependencies
     • Advanced dependencies like MPI have to be clarified, but should be determined by the users setup
     • The call to pip install could be customised for any module of choice (e.g. mixing with PyPI packages)

3. A lot of CMake 'magic' is involved for the reason of dune-py! It's there to ensure that it finds the right modules. I am not quite sure what can be simplified here, but it's not directly related to the venv setup, it's rather an orthogonal feature that requires a lot of CMake magic. We have to keep this in mind and have to collect ideas how to simplify this as well.

@samuel.burbulla good to hear from you!

As far as I know the internal venv is used and serves a different purpose which has no connection with the python bindings. The most important thing is that it has to be available during configuration since it is used during the cmake setup phase. That is at least what I understand. We perhaps don't need to install the Python bindings there but I'm not sure if someone needs Python bindings + the internal venv feature? Possibly for using bindings as part of C++ embedded Python this - no idea if someone is doing that at the moment.

As long as we don't reduce the multitude of ways people want to use Python I don't see a way of simplifying the setup.

Make the automatic venv setup optional per default (or remove it / move it to another module) [sorry Andreas, unpopular opinion, as especially you put so much work into this setup...]

I don't care at all about the internal venv - I don't embed Python or use the cmake-python features. So if we can rely on a user setup venv instead (installing required packages into that during dunecontrol) then that is fine with me. But others clearly wanted this feature. Is this feature only used in connection with dune-testtools? Could the venv setup be done there? Probably not or that would have been done...

I would even be happy with disabling Python support if dunecontrol is not run within an active venv but some don't want to setup a venv so that is not an option for opt-in/out either.

I don't care at all about the internal venv [...] But others clearly wanted this feature. Is this feature only used in connection with dune-testtools?

I think the issue was that dune-codegen installs several python packages with monkey patches (e.g. ufl), so installing them outside of the virtual environment would potentially screw other python packages unrelated to dune.

[...] but I'm not sure if someone needs Python bindings + the internal venv feature?

It would be a nice thing to someday have generated code from dune-codegen working with the bindings as well, but even with that ambitious wish I beleive that they could live in different environments because dune-codegen works more like a compiler for ufl.

One argument for having the internal-venv + bindings discussed at the time was to make it easier to have different cmake setups, i.e., a cmake with clang and one with gcc and corresponding venvs/dune-py for each. But that is perhaps not a good enough reason for the possible additional complexity.

But the automatic creation of a venv is not required for this splitting, is it? One could still create the venv manually and then tell DUNE (or cmake) about it. Or did I miss something?

That is correct and that is what is happening right now if you run/install DUNE in a previously created venv. I guess the wish was, that dune-codegen should be usable without a venv, because in the end it's a C++ module. But I'm not sure since I have never used this.

The question for me is, if this requires to manually switch the venv whenever using another build configuration or if it's possible to configure this once, such that each build persistently uses it's individual, pre-configured venv.

Creating and configuring one venv per build-configuration manually in advance would be fine. Having to switch manually would IMO be problematic.

You would have to switch manually - or add some hock to your folder so that a cd switches venvs.

For the bindings at least you would get an error messages saying that the venv is wrong if you try to run some python code. For the codegen/testtools setup it probably would not really matter since the packages in the venv there are I believe independent of the cmake configuration. @santiago.ospina, am I right in assuming that the same venv could be used with different cmake configurations in these use cases?

For the bindings at least this is only relevant for embedded python anyway since for python side scripts the cmake configuration used is defined by the venv active at the time the script is run (or at least the dune-py available). So the only way to use a different configuration is switching the environment.

I've spend a lot of effort to see how to use C++ sided embedded python with this framework with not too great a success because getting it to use the internal venv is not easy. Is anyone really using embndded Python with the bindings? It seemed always a would be great to have feature that nobody have really found a use for but I might just not know about it. So I'm a bit reluctant to make this possible future use and possibly then arising issues a show-stopper to reducing the complexity of the overall setup.

(declaimer) I can only argue from the c++ perspective - not much experience with python. So, If I say something obvious (or obviously wrong), I am sorry.

I am using different build folders for debug/release versions of my code (and for testing various compiler version). The build folder is typically not entered manually (e.g. using cd in the terminal), but everything is executed by cmake or dunecontrol from outside, e.g. cmake --build <build_dir> --target <target>. I don't know what would be the consequence for the python bindings if installed in a global venv, but we compile in two different build folders with different debug flags, for example. A manual action from the user whenever I compile something in a different build folder is easily forgotten. But, maybe I understand this procedure wrong.

I am not using embedded python (but also the bindings just for minimal experiments). I was hoping that these two things could be handled in an orthogonal way. But I have no idea how it is implemented.

Could the whole internal-venv, external-venv, no-venv,... setup somehow be abstracted out such that one could maintain the code of one configuration independent of the other (maybe this is already the case).

In general I am a fan of being explicit, with less "magic" (whatever this word means). E.g. in PETSc configuration it tries to find dependencies, if not found we get an error. Then we can either tell PETSc where to find it, or activate automatic download. But, a reasonable default action is always nice to have.

From the way I use the bindings: if I have different cmake configurations/dune-version I want to test I would have an external venv for each. Running a Python script using a given configuration then requires switching the venv and then running the script. Easy and foolproof (for me) and quite explicit (I need to run source ~/dune2.9-clang/bin/activate to use that setup). The only thing that could go wrong is changing something in one of the C++ sources and then running dunecontrol with the active dune2.9-clang environment although I previously used the 2.10 venv in the given builddir. But that would throws an error (or should) if I try to do that, i.e., run cmake with an active dune2.9-clang environment in a build folder that was previously configured within the dune2.10-gcc environment. If that check is not yet extensive enough that could probably be fixed.

@santiago.ospina, am I right in assuming that the same venv could be used with different cmake configurations in these use cases?

In general, I do not think so. If I understand correctly, the first installed module would constrain other modules to use its installed packages because other builds will try to pip install on top of that (I don't know what happens then). Of course that's fine if you rely on a registry, don't change your python packages, or if you have some meaningful versioning in place, but remember, your are pip installing python packages from source code, so anything can change at any time. In practice, the environment you choose is your "pythonic build environment" for whatever python source code you are developing so that they don't conflict with other builds. Same as with C++, you wouldn't like to mix them in the same folder. However, installing them is another story. Once you code is ready to be really installed into your system, you put your new shiny code into a public environment which is in general different from your "pythonic build environment" (currently, we install packages the raw python found by CMake, although I am not sure anyone ever used this).

My personal opinion is that the "pythonic build environments" should always be some sort of virtual environment orthogonal anything else in your system, including other builds. To be concrete, option 3 from @robert.kloefkorn is like building and modifying C++ applications directly in /usr/bin, which makes no sense to me.

Another argument against disabling virtual environments and still pip installing python packages is that building dune modules will potentially override the already installed dune module. You may remove your temporary build, yet, your permanent installed module will be broken forever.

Just to clarify what I believe @robert.kloefkorn (and I think @tkoch) are doing: no venv but also no package installation by configure. Instead @robert.kloefkorn sets his PYTHONPATH to the builddir/python folders of all dune modules - other packages like numpy etc. are installed into the system. In the current setup this works by not having an active external venv and disabling the setup of the internal venv through cmake flag which still sets up the Python package in the builddir but does not pip install anything. So @santiago.ospina issues with this approach I believe does not a problem since nothing is installed into the system.

That's a very interesting approach! If this currently works, one can simply install the build stage packages in a build directory (e.g. pip install --prefix <build-dir-path>) and append this path to the PYTHONPATH. This would be simplify the install logic because this is independent of whether your are in a venv.