cleaner concept for (non) distributed dof vectors
The following concerns mostly Lagrange DFs: There is a major issue with the expected state of a dof vector in different parts of dune-fem. It is not always clear if an operator for example expects a dof vector to be distributed (i.e. values on borders have to be summed up to get correct value of dof) and or if expects it to be non-distributed (i.e. the value of a dof is identical across all copies of that dof). There are a few issues
- it is not possible to find out the state of a given dof vector
- one can move from distributed to non distributed by calling
communicate
but gets disastrous results if one callscommunicate
twice - there is no way to go from non distributed to distributed
I want to suggest an extension to the interface (on the DiscreteFunction class). The names are first suggestions the concept is more important to me at the moment so please focus on that:
-
void clearSlaveDofs() const;
-
void gather() const;
-
void scatter() const;
-
bool isGathered() const;
-
void clear(setToState) const;
-
void forceState(newState) const;
-
the method acts as counterpart to
communicate
By settingslaveDofs = 0
a dof vector is switched into a "distributed" state souh.communicate(); uh.clearSlaveDofs(); uh.communicate();
works perfectly fine
2,3) The idea behind these methods is that a dof vector keeps track of its state and gather
will either call communicate (if the dof vector is in a distributed state) or do nothing. scatter
on the other hand will call clearSlaveDofs
if the dof vector was in a non distributed state and do nothing otherwise.
- is more for debugging purposes
- clears a vector and sets it into a given state (since a zero vector could be in either state).
- might be of interest when I know I have constructed a vector in a given state and just want to fix that state within the dof vector (but actually I would expect that
clear
will have been called so this might not be required).
The idea is that the actual user code never calls communicate
or clearSlaveDofs
but always gather
or scatter
. A rhs assembly for example could look like this:
void rhs(const DF &f, DF &u) {
f.gather();
u.clear(State::scatter);
for ... localU.axpy(...);
u.communicate(); // this could also be left out if the method is supposed to return in distributed state
}
If `rhs` is supposed to add to an existing `u` the call to `clear` could now be replaced by `u.scatter();`.
Note:
1) only `u.gather` is a non local operation but only if the state is distributed. Multiple calls to `gather`
don't cause a problem and don't cost time.
2) of course distributed has a lot to do with `functional` and non distributed with `function` but I wanted to avoid those slightly loaded terms
3) the whole proposal does not change any existing code (I think).
4) for a dg type space the semantics of these methods is not so clear to me - at the moment going from distributed to non distributed (i.e. calling `communicate`) involves a overwrite of all ghost dofs. `clearSlaveDofs` could simply be setting them all to zero I guess but leaving them unchanged would also work. One would actually expect that for a dof vector in distributed mode dofs on ghost elements should not be accessed at all so their value is irrelevant. For non distributed dof vectors the expectation would be that the ghost values are identical to the interior element on the other process.
One issue I'm not sure about how to address is the best implementation for the inverse operators. In principle I would like to be able to add some `gather/scatter` calls to the beginning of the inverse operators but of course the right state for `arg` and `dest` does depend on the operator being inverted. I think there is some unwritten assumption that a linear operator expects it's arguments to be distributed but will return the result non distributed? But perhaps this is a separate discussion to be had after the concepts given above have been added to dune-fem.