Installation seems to fail due to python as a normal user

I don't think I am doing something special.

While the installation returns success. the python stuff is not correctly installed.

I did reset the CMAKE_INSTALL_PREFIX to $HOME/dune_install and ran make install as a normal user:

-- Installing: /home/mblatt/dune_install/share/bash-completion/completions/dunecontrol
Installing python package (Debug)
CMake Warning at /home/mblatt/src/dune/opm-2.10/dune-common/cmake/modules/DuneExecuteProcess.cmake:76 (message):
  python package installation failed - ignored

  Run command:/usr/bin/cmake;--build;.;--target;install_python;--config;Debug

  Return code: 2

  Output:

  gmake[1]: Verzeichnis
  „/home/mblatt/src/dune/opm-2.10/dune-common/opm-parallel-debug“ wird
  betreten

  gmake[2]: Verzeichnis
  „/home/mblatt/src/dune/opm-2.10/dune-common/opm-parallel-debug“ wird
  betreten

  gmake[3]: Verzeichnis
  „/home/mblatt/src/dune/opm-2.10/dune-common/opm-parallel-debug“ wird
  betreten

  gmake[4]: Verzeichnis
  „/home/mblatt/src/dune/opm-2.10/dune-common/opm-parallel-debug“ wird
  betreten

  gmake[4]: Verzeichnis
  „/home/mblatt/src/dune/opm-2.10/dune-common/opm-parallel-debug“ wird
  verlassen

  gmake[4]: Verzeichnis
  „/home/mblatt/src/dune/opm-2.10/dune-common/opm-parallel-debug“ wird
  betreten

  Installing the python package at
  /home/mblatt/src/dune/opm-2.10/dune-common/opm-parallel-debug/python/.
  (location --user)

  error: externally-managed-environment

  

  × This environment is externally managed

  ╰─> To install Python packages system-wide, try apt install

      python3-xyz, where xyz is the package you are trying to
      install.
      
      If you wish to install a non-Debian-packaged Python package,
      create a virtual environment using python3 -m venv path/to/venv.
      Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
      sure you have python3-full installed.
      
      If you wish to install a non-Debian packaged Python application,
      it may be easiest to use pipx install xyz, which will manage a
      virtual environment for you. Make sure you have pipx installed.
      
      See /usr/share/doc/python3.11/README.venv for more information.

  

  note: If you believe this is a mistake, please contact your Python
  installation or OS distribution provider.  You can override this, at the
  risk of breaking your Python installation or OS, by passing
  --break-system-packages.

  hint: See PEP 668 for the detailed specification.

  gmake[4]: ***
  [python/CMakeFiles/install_python_package_017119b47aa3e31d2f1458e7ae21943a.dir/build.make:71:
  python/CMakeFiles/install_python_package_017119b47aa3e31d2f1458e7ae21943a]
  Fehler 1

  gmake[4]: Verzeichnis
  „/home/mblatt/src/dune/opm-2.10/dune-common/opm-parallel-debug“ wird
  verlassen

  gmake[3]: *** [CMakeFiles/Makefile2:5414:
  python/CMakeFiles/install_python_package_017119b47aa3e31d2f1458e7ae21943a.dir/all]
  Fehler 2

  gmake[3]: Verzeichnis
  „/home/mblatt/src/dune/opm-2.10/dune-common/opm-parallel-debug“ wird
  verlassen

  gmake[2]: *** [CMakeFiles/Makefile2:2040:
  CMakeFiles/install_python.dir/rule] Fehler 2

  gmake[2]: Verzeichnis
  „/home/mblatt/src/dune/opm-2.10/dune-common/opm-parallel-debug“ wird
  verlassen

  gmake[1]: *** [Makefile:621: install_python] Fehler 2

  gmake[1]: Verzeichnis
  „/home/mblatt/src/dune/opm-2.10/dune-common/opm-parallel-debug“ wird
  verlassen

Call Stack (most recent call first):
  python/cmake_install.cmake:55 (dune_execute_process)
  cmake_install.cmake:96 (include)


Installing wheel for python package at /home/mblatt/src/dune/opm-2.10/dune-common/opm-parallel-debug/python/. into /home/mblatt/dune_install/share/dune/wheelhouse...

Interestingly the return code of the install command is 0, like success.

mentioned in issue #381 (closed)

Note that I did not follow the Python discussions/implementations at all. Did not even read any install instructions. So this could very well be me being dumb. Maybe I am supposed to do something special.

This one might be Debian/Ubuntu specific. I faintly recall that it might be fixable by adding a specific flag or similar.

This is the command giving the error:

  cd /home/mblatt/src/dune/opm-2.10/dune-common/opm-parallel-debug/python &&
  /usr/bin/python3 -m pip install --user --upgrade
  --find-links=file:///home/mblatt/dune_install/share/dune/wheelhouse .

Yes, this is a debig-12 issue (probably also in some ubuntu versions) You are not allowed anymore to install in system directories. The advised way is to always install into a venv, or to pass --break-system-packages to the python command, I guess. I have no experience with installing the dune python packages. Others might be better in answering ideas how to properly pass this flag or create the venv before.

Just to clarify, this is not trying to install into any system directory. The --user flag means it gets installed into your local site-package. Apparently that is not allowed anymore but I have not looked into that because I always use a virtual env so don't get that. Errors in Python installations have always been Warnings and the Python packages just don't get configured. I think setting DUNE_PYTHON_INSTALL_LOCATION to none should solve the problem but I don't know if that is being tested.

The title says that installation fails but this should only be a warning. So that is a bug I guess but not sure why it is happening (it says CMake Warning at the top).

In my system it continues after the error message is shown. But it is actually an installation failure, since python packages are not installed. We need in the website or somewhere else a clear description what to do in this case. The error message states that you should activate a virtual env. But if you are not used to the dune-venv, you do not know which venv to activate or where to create one.

I have tried this out. When I create a venv (and activate it) after cmake+make was already run, after I got this install error, the error persists. When I rerun cmake+make it does not recognize the changed venv (before it was the default build-dir/dune-venv.) So, one has to do a full fresh build and delete everything. This is an information that needs to be given somewhere, at best before starting to build and install the python packages.

If someone want the bindings to be installed then they are using Python and should know how to setup a venv. Adding to the documentation that the cache needs to be removed if the Python setup changes is probably sensible. There is no way to avoid this since activating a venv could lead to a different Python version being used compared to the one found during the first configure.

The problem is (and this is what I would like to document somehow) that python bindings are enabled by default (this is not the problem), and this means that the default behavior if you do not do anything special is: make install results in an error (that is ignored but shown with a huge message)

My suggestion would be to disable (the installation of) python bindings if no vevn is used then all this goes away. Apparently installation into the user site-package is discouraged and so I would just disable this feature.

I also think that the warning is quite scary and always makes me wonder whether I made a mistake.

• One problem with the current warning is that is not actionable. It does not state clearly what is the problem. DUNE just says that it fails, and then debian proceeds to say how to fix the installation command, to which the user has no access to.

  • I know that installing system-wide is really bad, but one can still mirror the HUGE debian warning by suggesting to set the environmental variable PIP_BREAK_SYSTEM_PACKAGES=1. The user then can do it at its own risk, but at least the problem and its solution is clearly stated.

• Is there a way to know if pip install won't work before we even try it?

  • This way we can avoid the scary "error" messages with all the failed paths and show a HUGE warning stating the problem and how to solve it.

  • An alternative to this would be to simply not install system-wide by default when a venv was not found, and warn the user on installation that action needs to be taken in order to a system-wide installation of the python bindings: e.g. manually trigger the installation script:

    cmake -P <build-dir>/InstallDunePython.cmake

I respectfully disagree with "If someone want the bindings to be installed then they are using Python and should know how to setup a venv." I use Python regularly, but always for smaller things, and I guess that is not an usual usage pattern. Still, I have never needed to install anything using pip, because what Debian offered was always good enough for me. For the same reason, I have never needed to know what a venv is, until Dune forced me to.

As I wrote (and you cited) 'if someone wants the bindings installed' which is done with pip (we are not talking debian packaged dune here but source I think) so 'installation' requires pip and that requires a venv now apparently - so not sure what the disagreement is? If we are talking debian then of course my remark is wrong but then the installation should not involve 'pip --user' which is causing the error.

@oliver.sander I hope that with the next Debian release you can do that for DUNE, too. Let's hope it does not turn out to be very hard and I can find enough time.

An alternative to this would be to simply not install system-wide by default There is no 'system wide' installation by default at least there shouldn't be. How would that work anyway without sudo rights? We are using pip --user which installs into the $HOME/.local/lib/pythonX.Y/site-packages or something like that. The error message clearly states that

Installing the python package at
  /home/mblatt/src/dune/opm-2.10/dune-common/opm-parallel-debug/python/.
  (location --user)

@markus.blatt That would be great!

@andreas.dedner Thanks for the clarification!

mentioned in issue #382

changed title from Installation fails due to python as a normal user to Installation seems to fail due to python as a normal user

changed the description

I my intention, was to simply test this. I have not packaged the python stuff for Debian yet, but at least have the intention. So I am testing this outside of Debian currently to see how complicated it might be.

@andreas.dedner maybe you can help me a bit:

• If all python packages that a dune-module depends on are installed system-wide, what will be installed via pip?
• Is there a way to install just the generated python package of a dune module system-wide or do you see a manual way to do this?

Just to clarify - you really mean system wide now, i.e., with sudo or something? Is this for packaging? I don't really know much about that. pip can be used to do systemwide installations and I assume the usual versioning rules apply - so if the package in the system is in the required version range then nothing happens? But I really don't know

I mean if packages are found in the normal system path sys.path. On my machine with python3.11: '/usr/lib/python311.zip', '/usr/lib/python3.11', '/usr/lib/python3.11/lib-dynload', '/usr/local/lib/python3.11/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.11/dist-packages'

Yes this is for packaging.

Sounds good.

Does the warning actually go away with DUNE_PYTHON_INSTALL_LOCATION=none?

Yes, that works:

Installing wheel for python package at /home/mblatt/src/dune/opm-2.10/dune-common/opm-parallel-debug/python/. into /home/mblatt/dune_install/share/dune/wheelhouse...

Just not sure which location this should go to. I might have to tweak this a bit.

The wheels (which are some binary package) are apparently always installed and the location can be set by DUNE_PYTHON_WHEELHOUSE.

Perhaps those are the right objects for packaging anyway. But I never looked into wheels much - Dominic setup this part of the installation.

Just to clarify (for myself) what we are talking about:

1. User run make install not system wide and no active venv:

this would I believe currently install the python bindings and other required packages into the user site-package. That is not good practice apparently anymore.

Suggestion: remove this option and not install any Python packages at all in this scenario. That gets rid of the error/warning.

2. User run make install not system wide with active venv:

here Python packages can be easily installed into the venv which is current behavior

3. installation from pypi into a venv:

works as (2)

4. system wide installation from pypi:

have not tried that and don't know if there are issues with this

5. some system wide installation perhaps with apt or something:

this is outside of my experience and I have no idea how one does that

I am fine with deactivating installation of python bindings in the situation 1. It just would be good to have an information to the user about that, somewhere.