dlls and dylibs are not interchangeable- just like how you can’t use AudioUnits on Windows, you can’t use dlls on Mac. Every platform you want to target will need to be compiled for that platform separately, which means a .dll for Windows, a .dylib for Mac, a .so for Linux and Android, and various other formats for consoles that I can’t discuss on the public forums.
There is no Example VisualStudio solution for Mac, unfortunately the toolset is completely different and we have to use Xcode to compile anything for Mac, iOS and tvOS. You can open examples.xcworkspace in api/core/examples/xcode32 using Xcode and follow the steps listed in fmod_rnbo.cpp to build a .dylib version of your plugin.