Don't enable -Werror by default
The problem is that if someone is compiling with a newer version of gcc/clang than you are testing in your CI, they are likely to get new warnings, especially with a project of this size, and for an outside developer, having the build totally fail because of what is likely to be a harmless warning isn't terribly friendly.
Warnings do matter and you should definitely ensure that your code builds without warnings, but I would suggest leaving this option off by default and enabling it for your CI. Developers should know to fix all warnings before code is submitted and they can even manually enable this option for their local builds if they like, but it doesn't seem helpful to have the build fall over at the first hurdle over something trivial.