If you mean a VSTi, we currently have no support for them, VST is supported but limited to the platform the effect was created for.
If you mean custom instrument code or a library, you can create and FMOD DSP wrapper for it, which would allow you to add it in Studio as a ‘plug-in sound’. We ship some example plugins with our platform APIs you could use for reference.