The bootloader has its own copy of the stm32fxx library. The pre compiled library is just there for convenience since the core files are compiled with the Arduino IDE, which likes to rebuild everything often and that takes time.
The pre compiled library should have been compiled with the clock settings to run with a 10MHz quartz on an stm32f401. That includes HSE_VALUE and all PLL settings.
HSE_VALUE definition on core files it’s probably being ignored and I think it could be removed from compilation.
BRR settings on USARTs depend on PCLK frequency (PCLK2 for USART 1 and 6) and not directly from the system clock frequency. When configured correctly on our boards (PLL settings) PCLK2 runs at 84MHz, which is the max speed for APB2. I recommend you to use CubeMX to see the clock tree and get the right clock settings and values for your quartz.
Clocks are usually initialized by the very first lines of code, which in our boards is the bootloader. The bootloader calls SystemInit from startup_stm32f401xx.S. The possibility of skipping the bootloader is there, and doing that will call SystemInit from core files (which will point to the precompiled library at this point).
Long story short, you must either recompile the precompiled library, or you can modify how the core files are built and include your own stm32 library (sources, not precompiled).
I know it’s inconvenient but it just how the ST libraries are made. There should be a way to change the clock after but you don’t want to start up the system with the wrong settings. Also, HSE_VALUE is used for some peripheral settings like I2S.
I can check if there is project files for the precompiled library but I am not in the lab today.