setup-python.py fails 2

Running the Python bindings installation instructions (Second Approach):

./dune-common/bin/dunecontrol --opts=python.opts --module=dune-common all
./dune-common/bin/setup-dunepy.py --opts=python.opts --module=dune-common install

with an options file

CMAKE_FLAGS="-DBUILD_SHARED_LIBS=ON -DDUNE_ENABLE_PYTHONBINDINGS=ON \
             -DDUNE_PYTHON_INSTALL_LOCATION=user -DCMAKE_INSTALL_PREFIX=[...] -GNinja"
BUILDDIR="build-python"

results in a python error about parsing arguments:

Traceback (most recent call last):
  File "./dune-common/bin/setup-dunepy.py", line 131, in <module>
    main(sys.argv[1:])
  File "./dune-common/bin/setup-dunepy.py", line 66, in main
    key, value = arg.split('=', 1)
ValueError: not enough values to unpack (expected 2, got 1)
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 153, in apport_excepthook
    with os.fdopen(os.open(pr_filename,
FileNotFoundError: [Errno 2] No such file or directory: '/var/crash/_opt_sources_dune_dune-common_bin_setup-dunepy.py.1000.crash'

Original exception was:
Traceback (most recent call last):
  File "./dune-common/bin/setup-dunepy.py", line 131, in <module>
    main(sys.argv[1:])
  File "./dune-common/bin/setup-dunepy.py", line 66, in main
    key, value = arg.split('=', 1)
ValueError: not enough values to unpack (expected 2, got 1)

changed title from setup-python.py fails #2 to setup-python.py fails 2

The reason seems to be the -GNinja cmake parameter. This flag does not contain an = sign.

Note: cmake accepts lots of parameters. Many of them not simple variable defines, e.g.

-S <path-to-source>          = Explicitly specify a source directory.
-B <path-to-build>           = Explicitly specify a build directory.
-C <initial-cache>           = Pre-load a script to populate the cache.
-D <var>[:<type>]=<value>    = Create or update a cmake cache entry.
-U <globbing_expr>           = Remove matching entries from CMake cache.
-G <generator-name>          = Specify a build system generator.
-T <toolset-name>            = Specify toolset name if supported by
                               generator.
-A <platform-name>           = Specify platform name if supported by
                               generator.

And all those might be used in a CMAKE_FLAGS variable

Can you try with !918 (merged) ?

I updated the parameter parsing, but I think the while definitions dict is the wrong way to handle cmake arguments.

I don't know if this is used during pip install, that is why I didn't immediately change the interface here...

probably, one should use either a real argument parser, since we simply pass arguments to cmake, or maybe the whole variable should just be forwarded (no parsing of these flags at all)

!918 (merged) solves the direct argument parsing error. But I then get the next one:

Traceback (most recent call last):
  File "./dune-common/bin/setup-dunepy.py", line 136, in <module>
    main(sys.argv[1:])
  File "./dune-common/bin/setup-dunepy.py", line 102, in main
    duneModules = select_modules()
  File "/home/spraetor/dune-env/lib/python3.8/site-packages/dune/common/module.py", line 223, in select_modules
    modules = find_modules(get_module_path())
  File "/home/spraetor/dune-env/lib/python3.8/site-packages/dune/common/module.py", line 42, in find_modules
    description = Description(os.path.join(root, 'dune.module'))
  File "/home/spraetor/dune-env/lib/python3.8/site-packages/dune/packagemetadata.py", line 133, in __init__
    self.version = Version(data.get('version'))
  File "/home/spraetor/dune-env/lib/python3.8/site-packages/dune/packagemetadata.py", line 24, in __init__
    raise ValueError('Invalid version: \'' + s + '\'.')
ValueError: Invalid version: '1'.
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 153, in apport_excepthook
    with os.fdopen(os.open(pr_filename,
FileNotFoundError: [Errno 2] No such file or directory: '/var/crash/_opt_sources_dune_dune-common_bin_setup-dunepy.py.1000.crash'

Original exception was:
Traceback (most recent call last):
  File "./dune-common/bin/setup-dunepy.py", line 136, in <module>
    main(sys.argv[1:])
  File "./dune-common/bin/setup-dunepy.py", line 102, in main
    duneModules = select_modules()
  File "/home/spraetor/dune-env/lib/python3.8/site-packages/dune/common/module.py", line 223, in select_modules
    modules = find_modules(get_module_path())
  File "/home/spraetor/dune-env/lib/python3.8/site-packages/dune/common/module.py", line 42, in find_modules
    description = Description(os.path.join(root, 'dune.module'))
  File "/home/spraetor/dune-env/lib/python3.8/site-packages/dune/packagemetadata.py", line 133, in __init__
    self.version = Version(data.get('version'))
  File "/home/spraetor/dune-env/lib/python3.8/site-packages/dune/packagemetadata.py", line 24, in __init__
    raise ValueError('Invalid version: \'' + s + '\'.')
ValueError: Invalid version: '1'.

I agree, we should not try to be clever with the flags. I'll have a look, but overall I'd prefer to only have a string for the parameters, or a list of parameters, where -DCMAKE_FOO=BAR would be only single entry.

IMHO the dict is the wrong interface here, as we have so many different potential entries.

I'll try something in the MR.

OK. Keeping everything as one string will not work, as cmake is called vis subprocess.Popen, which expects an argument list. Still we can handle all arguments as arguments and not definitions.

I updated !918 (merged) to store the cmake arguments a list instead of a dict.

@simon.praetorius can you check whether this fixes you problem?

This is a general problem.

Our version constraints differ significantly from those posed by python.

mentioned in commit ead43f11

mentioned in commit 5ee6c45d

mentioned in commit f7085563

mentioned in commit 5fccea67

closed with merge request !918 (merged)

mentioned in commit 9a4f79ad