Fix scope of {fmt} library in CMake

The common headers are using {fmt}, if their headers are placed somewhere else than the system headers, the current set up would fail to find the headers.