I need a granular synthesizer for vehicle engine audio, but so far have been given the runaround by literally anyone and everyone with a solution for this that works with FMOD (AudioMotors 2 is defunct, and REV hasn’t responded to any of my queries. Other community members with similar solutions have ghosted me as well), so I’m locked into figuring this out myself. Unfortunately for me, a DSP plugin for FMOD Studio seems out of reach as I do not know C++ at all.
I’ve been trying to hack something together using programmer instruments and audio tables, but haven’t been able to produce seamless grain playback.
I’ve taken a look at the example granular_synth.cpp project, and tried to replicate it using FMOD Studio API calls in Unity C# using a programmer instrument, but Studio seems to lack any way to call setDelay on an event so that things are queued up correctly. (You can technically get the channel to call set delay on, but it produces the ERR_STUDIO_NOT_LOADED error until the next FMOD update as I understand it, which is too late)
I’ve also been reading that you shouldn’t mix Studio and Core api calls together, so I’m kind of at a loss here on what to do.
I think I read that it is possible to build DSP plugins for Studio and whatnot using C# and not just C++ but there is literally no documentation on how this would work as far as actually setting up and building a project goes.
We have an example of how to set up what is effectively your own granular synthesizer for engine audio in the Vehicles/Car Engine event in the Examples project that ships with FMOD Studio.
This solution doesn’t rely on code, and is a common solution for engine sound design.
If you prefer writing your own granular synthesizer, it is certainly possible to create a DSP in C# and we have an example of how to implement a DSP here Scripting Examples | DSP Capture.
In this case, the “plugin” isn’t a dynamic library as it would be with REV or AudioMotors, it is just C# code that implements an FMOD.DSP_DESCRIPTION.
Most of the fields are optional, so to create a minimal DSP you just need to implement numinputbuffers, numoutputbuffers, and a read callback. Here is a simple DSP written in C# that just generates white noise:
[AOT.MonoPInvokeCallback(typeof(FMOD.DSP_READ_CALLBACK))]
static FMOD.RESULT ReadCallback(ref FMOD.DSP_STATE dsp_state, IntPtr inbuffer, IntPtr outbuffer, uint length, int inchannels, ref int outchannels)
{
System.Random r = new System.Random();
int len = (int)length * inchannels;
float[] buffer = new float[len];
for (int s = 0; s < len; s++)
{
buffer[s] = 0.1f * (float)r.NextDouble();
}
Marshal.Copy(buffer, 0, outbuffer, len);
return FMOD.RESULT.OK;
}
void Start()
{
// Assign the callback to a member variable to avoid garbage collection
mReadCallback = ReadCallback;
FMOD.DSP_DESCRIPTION desc = new FMOD.DSP_DESCRIPTION();
desc.numinputbuffers = 2;
desc.numoutputbuffers = 2;
desc.read = mReadCallback;
FMODUnity.RuntimeManager.CoreSystem.getMasterChannelGroup(out FMOD.ChannelGroup masterCG);
FMODUnity.RuntimeManager.CoreSystem.createDSP(ref desc, out mCaptureDSP);
masterCG.addDSP(0, mDSP);
}
void OnDestroy()
{
FMODUnity.RuntimeManager.CoreSystem.getMasterChannelGroup(out FMOD.ChannelGroup masterCG);
masterCG.removeDSP(mDSP);
mCaptureDSP.release();
}
As for an actual granular synthesis implementation, getting something better than REV or AudioMotors is a pretty ambitious goal, but I would do something like:
Decode an audio file into memory
Pick a couple of random positions in the file to playback from
Copy data from each of those positions for some random number of samples
Fade in and out each of the “grains” created above to prevent clicks
Repeat from step 2 every couple of Read callback invocations, creating new grains as time goes on
Hi Jeff. First of all, thank you so much for taking the time to write this all out for me.
So your first example is what I’ve been doing so far, but it relies on a set of really good engine loops at various RPMs, which are very difficult to find and also tend to lack the “bite” of an engine under load, which is what I’m looking for.
As for REV and AudioMotors, I’m not necessarily looking to better them, just making something using the underlying tech to get me started since it’s been so difficult to get hold of them. I recently got an email back from the FMOD sales team though saying AudioMotors ownership has been transferred to FMOD and will be continued this way, so that bodes well for maybe just using that though.
As for writing a DSP in C#, this is incredibly helpful. Though maybe I’m just not versed enough and need to read through the documentation more, but this would just play the audio directly “to speaker” so-to-speak right? Can I still spatialize and apply an effect chain to this like through an event?
Yeah this DSP won’t be spatialized. You have a few options for spatializing the output of a DSP, the simplest would be to attach the DSP to a 3D sound.
You will need to update the position of the Channel object (essentially an instance of a Sound object) as well with Channel.set3DAttributes.
private FMOD.Sound mSound;
private FMOD.Channel mChannel;
private FMOD.CREATESOUNDEXINFO mExinfo = new FMOD.CREATESOUNDEXINFO();
void Start()
{
mReadCallback = ReadCallback;
FMOD.DSP_DESCRIPTION desc = new FMOD.DSP_DESCRIPTION();
desc.numinputbuffers = 2;
desc.numoutputbuffers = 2;
desc.read = mReadCallback;
// Exinfo required for playing a custom "user" sound
mExinfo.cbsize = Marshal.SizeOf(typeof(FMOD.CREATESOUNDEXINFO));
mExinfo.numchannels = 2;
mExinfo.defaultfrequency = 48000;
mExinfo.length = 96000;
mExinfo.format = FMOD.SOUND_FORMAT.PCMFLOAT;
FMODUnity.RuntimeManager.CoreSystem.getMasterChannelGroup(out FMOD.ChannelGroup masterCG);
FMODUnity.RuntimeManager.CoreSystem.createSound("", FMOD.MODE._3D | FMOD.MODE.LOOP_NORMAL | FMOD.MODE.OPENUSER, ref mExinfo, out mSound);
FMODUnity.RuntimeManager.CoreSystem.createDSP(ref desc, out mCaptureDSP);
FMODUnity.RuntimeManager.CoreSystem.playSound(mSound, masterCG, false, out mChannel);
// Add the DSP to the sound's FMOD.Channel instead of the master channel group
mChannel.addDSP(FMOD.CHANNELCONTROL_DSP_INDEX.TAIL, mDSP);
}
void OnDestroy()
{
mChannel.removeDSP(mDSP);
}
void Update()
{
var attr = FMODUnity.RuntimeUtils.To3DAttributes(gameObject);
mChannel.set3DAttributes(ref attr.position, ref attr.velocity);
}
If you wanted to apply an effect chain from FMOD Studio you could pass this custom sound through a programmer sound callback, which would also allow you to handle spatialization through the event.
Oh I can use this with a programmer sound callback? I had read that it wasn’t a good idea to mix Studio and Core level callbacks. I can try giving it a shot though.
Hey again, I’ve only been able to try this out just now, but I think I’m in over my head a little.
I’ve got individual grains in an audio table, and I’m able to parse a string to select the appropriate grain and create a sound from it. I’m now struggling to actually transfer that sound date to the dsp callback so it can play it.
Maybe I need to queue the data up to a buffer first or something, but I’m not sure. I’ve figured out how to send the sound as userdata but after that I’m not sure. I tried using @lock on it but I feel like this is completely wrong.
If you have gotten the FMOD.Sound into the user data of a DSP read callback then you can read the raw PCM data with Sound.readData. Here is an example on Extracting PCM Data From a Sound.
Once you get the pcm data out, you just need to copy the data into the out buffer during the callback. Using the ScriptUsageDspCapture.cs example again that would be:
private float[] mDataBuffer;
private float[] mPreloadedSound;
private int mReadPos = 0;
[AOT.MonoPInvokeCallback(typeof(FMOD.DSP_READ_CALLBACK))]
static FMOD.RESULT CaptureDSPReadCallback(ref FMOD.DSP_STATE dsp_state, IntPtr inbuffer, IntPtr outbuffer, uint length, int inchannels, ref int outchannels)
{
FMOD.DSP_STATE_FUNCTIONS functions = (FMOD.DSP_STATE_FUNCTIONS)Marshal.PtrToStructure(dsp_state.functions, typeof(FMOD.DSP_STATE_FUNCTIONS));
IntPtr userData;
functions.getuserdata(ref dsp_state, out userData);
GCHandle objHandle = GCHandle.FromIntPtr(userData);
ScriptUsageDspCapture obj = objHandle.Target as ScriptUsageDspCapture;
// Save the channel count out for the update function
obj.mChannels = inchannels;
// Don't read outside buffer length
if (obj.mReadPos > obj.mPreloadedSound.Length)
{
obj.mReadPos = 0;
}
// Copy the preloaded sound into the display
int lengthElements = (int)length * outchannels;
Array.Copy(obj.mPreloadedSound, obj.mReadPos, obj.mDataBuffer, 0, lengthElements);
// Copy the preloaded sound into the out buffer
Marshal.Copy(obj.mPreloadedSound, obj.mReadPos, outbuffer, lengthElements);
obj.mReadPos += lengthElements;
return FMOD.RESULT.OK;
}
void Start()
{
// Assign the callback to a member variable to avoid garbage collection
mReadCallback = CaptureDSPReadCallback;
FMODUnity.RuntimeManager.CoreSystem.createSound(Application.streamingAssetsPath + "/your_source_file.wav", FMOD.MODE.OPENONLY, out FMOD.Sound sound);
sound.getLength(out uint length, FMOD.TIMEUNIT.PCMBYTES);
byte[] buffer = new byte[(int)length];
sound.readData(buffer);
// Convert the byte array to a float array for displaying and playing back in the callback
mPreloadedSound = new float[length / sizeof(float)];
for (int i = 0; i < mPreloadedSound.Length; ++i)
{
mPreloadedSound[i] = BitConverter.ToSingle(buffer, i * sizeof(float));
}
...
}
So I think I’ve got it all set up, but now the mSound.getLength call is returning 0 even though its loaded a new sound. I’m creating a sound based off of an index/code from an audio table. I’ve confirmed this works when loaded into a normal programmer event… the sound is indeed working and playing there, but using this method it doesn’t give me a length. See below for my whole code so far: relevant bit is in Start()
public class GranularSynthEngine : MonoBehaviour
{
private const string ACCEL_KEY = "Accel";
private const string DECEL_KEY = "Decel";
private const int MAX_RPM = 10000;
[Range(0, MAX_RPM)]
public float rpm;
public float Progress => rpm / MAX_RPM;
[SerializeField] private string baseKey = "Anarchy";
[SerializeField] private int accelGrainCount = 82;
[SerializeField] private int decelGrainCount = 57;
[SerializeField] private bool isDecelerating;
private int _currentGrainIndex;
private int _sampleRate;
private string AccelKey => isDecelerating ? DECEL_KEY : ACCEL_KEY;
private string CurrentKey => $"{baseKey}_{AccelKey}_{_currentGrainIndex:000}";
private FMOD.DSP_READ_CALLBACK mReadCallback;
private FMOD.DSP mCaptureDSP;
private GCHandle mObjHandle;
private uint mBufferLength;
private FMOD.Sound mSound;
private FMOD.Channel mChannel;
private float[] _preloadedSound;
private int _readPos = 0;
void Start()
{
mReadCallback = CaptureDSPReadCallback;
// Get a handle to this object to pass into the callback
mObjHandle = GCHandle.Alloc(this);
if (mObjHandle != null)
{
DSP_DESCRIPTION desc = new DSP_DESCRIPTION();
desc.numinputbuffers = 1;
desc.numoutputbuffers = 1;
desc.read = mReadCallback;
desc.userdata = GCHandle.ToIntPtr(mObjHandle);
RuntimeManager.StudioSystem.getSoundInfo(CurrentKey, out var soundInfo);
// Exinfo required for playing a custom "user" sound
soundInfo.exinfo.cbsize = Marshal.SizeOf(typeof(CREATESOUNDEXINFO));
soundInfo.exinfo.numchannels = 1;
soundInfo.exinfo.defaultfrequency = 48000;
soundInfo.exinfo.length = 96000;
soundInfo.exinfo.format = SOUND_FORMAT.PCMFLOAT;
MODE soundMode = MODE._3D | MODE.LOOP_NORMAL | MODE.OPENUSER;
var result = RuntimeManager.CoreSystem.createSound(soundInfo.name_or_data, soundMode | soundInfo.mode, ref soundInfo.exinfo, out mSound);
if (result != FMOD.RESULT.OK)
{
Debug.LogError($"Failed to create sound: {result}");
return;
}
RuntimeManager.CoreSystem.getMasterChannelGroup(out ChannelGroup masterCG);
RuntimeManager.CoreSystem.createDSP(ref desc, out mCaptureDSP);
RuntimeManager.CoreSystem.playSound(mSound, masterCG, false, out mChannel);
// Add the DSP to the sound's FMOD.Channel instead of the master channel group
mChannel.addDSP(CHANNELCONTROL_DSP_INDEX.TAIL, mCaptureDSP);
mSound.getLength(out uint length, TIMEUNIT.PCMBYTES);
byte[] buffer = new byte[(int)length];
mSound.readData(buffer);
// Convert the byte array to a float array for displaying and playing back in the callback
_preloadedSound = new float[length / sizeof(float)];
for (int i = 0; i < _preloadedSound.Length; i++)
{
_preloadedSound[i] = BitConverter.ToSingle(buffer, i * sizeof(float));
}
}
}
void OnDestroy()
{
Cleanup();
}
public void Cleanup()
{
mChannel.removeDSP(mCaptureDSP);
mCaptureDSP.release();
}
void Update()
{
var attr = FMODUnity.RuntimeUtils.To3DAttributes(gameObject);
mChannel.set3DAttributes(ref attr.position, ref attr.velocity);
// MODE soundMode = MODE.LOOP_NORMAL | MODE.CREATECOMPRESSEDSAMPLE | MODE.NONBLOCKING;
// RuntimeManager.StudioSystem.getSoundInfo(CurrentKey, out var soundInfo);
// RuntimeManager.CoreSystem.createSound(soundInfo.name_or_data, soundMode | soundInfo.mode, ref soundInfo.exinfo, out mSound);
}
[AOT.MonoPInvokeCallback(typeof(FMOD.DSP_READ_CALLBACK))]
static FMOD.RESULT CaptureDSPReadCallback(ref FMOD.DSP_STATE dsp_state, IntPtr inbuffer, IntPtr outbuffer, uint length, int inchannels, ref int outchannels)
{
FMOD.DSP_STATE_FUNCTIONS functions = (FMOD.DSP_STATE_FUNCTIONS)Marshal.PtrToStructure(dsp_state.functions, typeof(FMOD.DSP_STATE_FUNCTIONS));
IntPtr userData;
functions.getuserdata(ref dsp_state, out userData);
GCHandle objHandle = GCHandle.FromIntPtr(userData);
GranularSynthEngine synthEngine = objHandle.Target as GranularSynthEngine;
if (synthEngine == null)
{
Debug.LogError("Failed to get GranularSynthEngine from GCHandle");
return FMOD.RESULT.ERR_INVALID_PARAM;
}
int lengthElements = (int)length / outchannels;
try
{
// Don't read outside the buffer length, loop
if (synthEngine._readPos + lengthElements > synthEngine._preloadedSound.Length)
{
// Read buffer in two parts
int firstLength = synthEngine._preloadedSound.Length - synthEngine._readPos;
Marshal.Copy(synthEngine._preloadedSound, synthEngine._readPos, outbuffer, firstLength);
synthEngine._readPos += firstLength;
int secondLength = lengthElements - firstLength;
Marshal.Copy(synthEngine._preloadedSound, 0, outbuffer + firstLength, secondLength);
synthEngine._readPos = secondLength;
}
else
{
// Read whole buffer
Marshal.Copy(synthEngine._preloadedSound, synthEngine._readPos, outbuffer, lengthElements);
synthEngine._readPos += lengthElements;
}
}
catch (Exception e)
{
Debug.LogError(e.Message);
synthEngine.Cleanup();
throw;
}
return FMOD.RESULT.OK;
}
}
