Feature/scikit build

We have developed a possibility to distribute DUNE modules as python packages. With that, any DUNE module can be distributed and installed via pip. This includes modules that do not provide python bindings of their own.

Core modules

Versions of some core modules are already uploaded to PyPI. For instance, you can install dune-grid (together with dune-common and dune-geometry) simply (preferably in a virtual environment) via

pip install dune-grid

Then, you should be able to directly use the python bindings, e.g.

from dune.grid import structuredGrid
grid = structuredGrid([0,0], [1,1], [100,100])
print(grid.size(0))

All core modules can be installed using

pip install dune-grid dune-istl dune-localfunctions

A more complete example script is available when running

python -m dune.grid

and looking into the newly generated grid_tutorial folder.

C++ developers

Note that the scripts dunecontrol and duneproject are available and the installed core modules are correctly found in the activated virtual environment.

Adding support to existing modules

It is quite easy to add support for packaging existing modules for uploading to PyPi.

Preparing the module

To provide sufficient metadata for the package additional field should be added to dune.module. These fields are:

Author: S. Body
Description: Some dune mdoule
URL: e.g. gitlab link
Python-Requires: numpy scipy

The last line is only required in case Python bindings are available that require additional packages to be installed. Here the usual Python syntax can be used, e.g., numpy==1.19.4. The string for the description entry can be taken from the pc.in file and there replaced with the line

Description: @DESCRIPTION@

To generate the distribution package run

PATH_TO_DUNE_COMMON/bin/dunepackaging.py --onlysdist

this should generate the files setup.py, pyproject.toml and the folder dist containing the tar.gz file ready for upload to pypi.

Using dunecontrol packages can be build an uploaded for all available source repos by executing

PATH_TO_DUNE_COMMON/bin/dunecontrol exec PATH_TO_DUNE_COMMON/bin/dunepackaging.py --onlysdist

The version string can be changed by providing a --version parameter to dunepackaging.py.

Testing

Local testing of the package can be done using

pip install -v --log logfile --find-links file://$PWD/dist dune-foo

where dist is a folder containing all the generated dist files. Note that the wheel cache is used (and is crucial for efficient building of the modules) so removing ~/.cache/pip can be required to force pip to use the new packages.

Uploading

If you have an account on PyPi and an active token simply running

PATH_TO_DUNE_COMMON/bin/dunepackaging.py --upload pypi

will build the package and upload to pypi. Other repositories can be provided: testpypi and gitlab - like pypi each requires an entry in ~/.pypi.rc.

Versioning

The version number specification is parsed from the dune.module file or can be passed as argument --version to the dunepackaging script. If the version specifier contains a -git postfix (e.g. 2.8-git) the postfix is removed and the major.minor.revision version specifier (2.8.0) is appended by .devDATE where DATE is the current day in YYYYMMDD format (2.8.0.dev31012020).

Dependencies

There are two kinds of dependencies of python packages:

1. Build time dependency (in pyproject.toml)
2. Run time dependency (in setup.py's install_requires)

The modules given in the dune.module file are used as build/run time dependency as follows:

dune.module	pyproject.toml	setup.py
Depends	x	x
Suggests
Python-Requires:	x	x

Versioning of dependencies

If the version specifier contains a -git postfix or a version number was passed as command line argument, the version number is set as maximum version requirement (<=) for all dependent dune modules. This ensures that a published version of a module will not be invalidated by an updated version of some dependency.

More information

We use scikit-build as glue between the setuptools Python module and CMake.