[math] Use integer type in Factorial

Using an enum has the side effect that you can't do arithmetic without explicitly casting to an integer type. This also produced a compiler warning with clang 10 "arithmetic between different enumeration types is deprecated" -Wdeprecated-anon-enum-enum-conversion