To install FMOD to the engine instead of the project folder, in Windows, you need to create a Marketplace folder within Plugins and put the FMODStudio plugin folder in there. It should read something like this:
C:\Program Files\Epic Games\UE_5.0\Engine\Plugins\Marketplace\FMODStudio (For Windows)
Whereas in UE4 it would’ve been:
C:\Program Files\Epic Games\UE_5.0\Engine\Plugins\FMODStudio
Apparently Unreal 5 is looking for 3rd Party Engine plugins in the Marketplace folder instead of the Plugins folder.
For the source code version, you can put the plugin in the usual Engine/Plugins/FMODStudio folder and UnrealBuildTool will find it and compile it with no issues.
For the pre-compiled version, it doesn’t matter where you put the plugin folder. The engine can find the .uplugin but it can’t find the module for it. So you either have to compile from source or put it into your Project/Plugins folder. I have yet to find a way to get the plugin working by having the folder somewhere within the engine folder.
So far this works in Unreal 5.0. I tried it with Unreal 5.1 but the FMOD Plugin was built using the preview version of 5.1. Will have to wait for the Public Release version of 5.1 to come out.