Hi,
Could I grab your FMOD Version? Would it be possible to share the code so I can try and reproduce the issue on my side? It can be uploaded to your profile.
We also have an Core API example for writing output to a .mp3 file: "C:\Program Files (x86)\FMOD SoundSystem\FMOD Studio API Windows\api\core\examples\plugins\output_mp3.cpp" which may have some useful information. Rather than using a custom DSP you can create a custom output plugin: FMOD Engine | Core Api System - System::Setoutputbyplugin.
The mixer takes 10ms~ to update so you could try only calling the system->update() only after 10ms? (Abnormal sound when using FMOD_OPENUSER - #4 by jeff_fmod).