Is this maybe a false positive from build-log scanning? This is a header-only library and installed packages contain only headers, CMake and pkg-config files, and the latter do not appear to set -march=native as a required flag for downstream compilation.
-march=native is set when compiling the test suite (test/CMakeLists.txt), which could be patched out, but since those binaries shouldn't leave the build machine, does it matter?