Revert "Add a few explicit casts to silence warnings concerning overloading ambiguities"

This reverts commit cf4bf4d8.

The epxlicit cast isn't needed anymore as we have a fix for the
ambiguous operator overload.