Greetings,
I’ve been pulling my hair on trying to find any reference on how to get the time domain data from a sound. I’ve seen several instance where channel::getWaveData claim to do just that but it seems to be non-existent from the latest API version.
Is there any example available that I could have missed? How does channel::getWaveData translate to FMOD 5?
Thank you,
check the dsp_custom example in the FMOD Studio low level examples. You can capture PCM wavedata yourself through a callback.
getWaveData only exists in FMOD Ex.
This is described in the documentation http://www.fmod.org/documentation/#content/generated/overview/transitioning.html
Ok, the custom DSP is in place would you mind elaborate on how do we get the time domain of the playing sound exactly?
Should we be concern about retrieving spectral data (FMOD_DSP_FFT_SPECTRUMDATA) in on loop and the wave data in another?
I’ve been trying to figure how to get the wave data but I’m in the dark.
I guess my first question would be, why do we have to make a custom DPS in the first place? What about having the same mechanic as FMOD_DSP_FFT_SPECTRUMDATA like FMOD_DSP_FFT_WAVEDATA?!
When using a custom DSP, how do we get the callback data from the main thread?
Is there any documentation or example that could be expected in a near future on how to retrieve wave data with FMOD 5?
look in the dsp_custom example.
http://www.fmod.org/documentation/#content/generated/example_lowlevel/dsp_custom.html
There are examples of dsps in the example folder in a few places.
If you write PCM data to a shared buffer in memory, you simply read from it from the main thread. If you want to avoid tearing you can lock the read against the write with a mutex.
Hi Brett, thank you for your input.
I hear you but there is a lot of figuring out without a written example. While this might sound very clear to most, writing PCM data is already ambiguous to me. Do you mean the in/out buffer of a custom DSP? What is the in/out buffer return anyways? How exactly do you share the memory with the custom DSP and the main thread? DSP::setParameterData and DSP::getParameterData ? I’ve looked at all the examples and none of them seems to shared a buffer in memory.
The custom dsp code is pretty much all you need to look at.
You can see it passes input signal (inbuffer) to the output signal (outbuffer). You always have to do that if you want to hear sound. In custom_dsp’s case it is scaling the input. Just change that to no scaling! (remove the multiply)
Next , you just want to copy the same input buffer to a global memory buffer. You dont need to complicate it by using setParameter/getParameter, just allocate a buffer, write to it from the callback, and read from it in your drawing thread (ie main thread).
This assumes that you know how to allocate heap memory as a global variable.
It also assumes you know how to parse interleaved PCM data (ie is the incoming signal stereo? Then you have to parse it as such).
http://www.fmod.org/documentation/#content/generated/overview/terminology.html has more information on this.
Hi Brett, thank you for your support in this endeavor. Global variables would be on very last resort. I’m expecting to have several sounds with their own custom DSP simultaneously. I’d be interested in using setParameter/getParameter you referred to, would you mind to elaborate? I don’t have any problem using mutex lock/unlock as well if needed. At this point, I just need to know how to pass the data, for a lack a of better word, elegantly. What is the custom DSP out buffer UNITS exactly? Is it decibels?
Hi,
You can use setUserData on a DSP and getUserData inside the dsp read callback, which is easier, or you’ll have to define callbacks for setParameterData/getParameterData if you want to do that. The implementation of that is varied, you could set and get the address of a buffer, or just simply allocate the buffer in an init callback and use getParameterData to get it (from the app) if you wanted.
the buffer is just linear floating point data, from -1.0 to +1.0.
I still don’t get how to pass the out buffer to the DSP, here is some pseudo code that could help clear this out:
FMOD_RESULT F_CALLBACK TimeDomainCallback(
FMOD_DSP_STATE *dsp_state,
Float *in_buffer,
Float *out_buffer,
UInt in_length, Int in_channels, Int *out_channels)
{
memcpy(out_buffer, in_buffer, in_length * in_channels * sizeof(Float));
???->setUserData(out_buffer);
return FMOD_OK;
}
void ZFmodSoundSpectrum::Init()
{
FMOD_DSP_DESCRIPTION l_desc;
memset(&l_desc, 0, sizeof(l_desc));
l_desc.version = 0x00010000;
l_desc.numinputbuffers = 1;
l_desc.numoutputbuffers = 1;
l_desc.read = TimeDomainCallback;
ZFmod::GetSystem()->createDSP(&l_desc, &m_DspTime);
}
void ZFmodSoundSpectrum::Update()
{
Float l_buffer;
m_DspTime->getUserData(l_buffer);
}
What am I missing?
You are setting the userdata as the address of a temporary variable. You can’t do that.
Next if you dont know how to get the handle of the dsp, you should look at the state that was passed into your callback. https://www.fmod.org/docs/content/generated/FMOD_DSP_STATE.html
the DSP instance is there.
Also in your case you’d probably be more interested in FMOD_DSP_STATE::plugindata
If you set up a ‘create’ callback, your create callback could allocate the secondary buffer, and the read callback could write to it, and your getParameterData callback could return it to the user for reading.
If you set up a close callback as well, you can free the memory.
What is the linear floating point represents?
The amplitude of the signal? Its just a representation of the waveform. There is no decibel range because the loudness of the signal is not defined. I’m not sure how else to explain it , besides going to this wikipedia article Pulse-code modulation - Wikipedia
Thank you for your suggestion Brett but all I’m looking for is the replacement of channel::getWaveData. For all I could gather from your explanations, It seems overly complicated and there is no universe where I can figure this out without some piece of code.
As an example, here is how web audio is doing it:
One line of code, analyser.getByteTimeDomainData(dataArray); a working example and 30 seconds later, you’re up and running.
I’m still hopping for a solution but it is mind boggling that FMOD removed getWaveData without any concrete alternative.
Thanks anyways,
Hi,
We can add some documentation on how to do it, but i’ve already pretty much given you the answer in the examples and posts above. getWaveData gives you a non continuous window of data, and a dsp callback is a function that gives you the same data, but in a continuous window. All you have to do is copy it from the callback’s inbuffer pointer to the same buffer you would have used as with getWaveData. There is barely extra work in doing this.
Hi Brett, thank you for your input. I see the value in this. If it is not too much to ask, would you mind sharing a code snippet on how to pass the out buffer from the call back to the main thread (not using global variables)?
Here is a modification of the dsp_custom example that prints out some of the data in realtime
/*==============================================================================
Custom DSP Example
Copyright (c), Firelight Technologies Pty, Ltd 2004-2017.
This example shows how to add a user created DSP callback to process audio
data. The read callback is executed at runtime, and can be added anywhere in
the DSP network.
==============================================================================*/
#include "fmod.hpp"
#include "common.h"
typedef struct
{
float *buffer;
float volume_linear;
int length_samples;
int channels;
} mydsp_data_t;
FMOD_RESULT F_CALLBACK myDSPCallback(FMOD_DSP_STATE *dsp_state, float *inbuffer, float *outbuffer, unsigned int length, int inchannels, int *outchannels)
{
mydsp_data_t *data = (mydsp_data_t *)dsp_state->plugindata;
/*
This loop assumes inchannels = outchannels, which it will be if the DSP is created with '0'
as the number of channels in FMOD_DSP_DESCRIPTION.
Specifying an actual channel count will mean you have to take care of any number of channels coming in,
but outputting the number of channels specified. Generally it is best to keep the channel
count at 0 for maximum compatibility.
*/
for (unsigned int samp = 0; samp < length; samp++)
{
/*
Feel free to unroll this.
*/
for (int chan = 0; chan < *outchannels; chan++)
{
/*
This DSP filter just halves the volume!
Input is modified, and sent to output.
*/
data->buffer[(samp * *outchannels) + chan] = outbuffer[(samp * inchannels) + chan] = inbuffer[(samp * inchannels) + chan] * data->volume_linear;
}
}
data->channels = inchannels;
return FMOD_OK;
}
/*
Callback called when DSP is created. This implementation creates a structure which is attached to the dsp state's 'plugindata' member.
*/
FMOD_RESULT F_CALLBACK myDSPCreateCallback(FMOD_DSP_STATE *dsp_state)
{
unsigned int blocksize;
FMOD_RESULT result;
result = dsp_state->functions->getblocksize(dsp_state, &blocksize);
ERRCHECK(result);
mydsp_data_t *data = (mydsp_data_t *)calloc(sizeof(mydsp_data_t), 1);
if (!data)
{
return FMOD_ERR_MEMORY;
}
dsp_state->plugindata = data;
data->volume_linear = 1.0f;
data->length_samples = blocksize;
data->buffer = (float *)malloc(blocksize * 8 * sizeof(float)); // *8 = maximum size allowing room for 7.1. Could ask dsp_state->functions->getspeakermode for the right speakermode to get real speaker count.
if (!data->buffer)
{
return FMOD_ERR_MEMORY;
}
return FMOD_OK;
}
/*
Callback called when DSP is destroyed. The memory allocated in the create callback can be freed here.
*/
FMOD_RESULT F_CALLBACK myDSPReleaseCallback(FMOD_DSP_STATE *dsp_state)
{
if (dsp_state->plugindata)
{
mydsp_data_t *data = (mydsp_data_t *)dsp_state->plugindata;
if (data->buffer)
{
free(data->buffer);
}
free(data);
}
return FMOD_OK;
}
/*
Callback called when DSP::getParameterData is called. This returns a pointer to the raw floating point PCM data.
We have set up 'parameter 0' to be the data parameter, so it checks to make sure the passed in index is 0, and nothing else.
*/
FMOD_RESULT F_CALLBACK myDSPGetParameterDataCallback(FMOD_DSP_STATE *dsp_state, int index, void **data, unsigned int *length, char *)
{
if (index == 0)
{
unsigned int blocksize;
FMOD_RESULT result;
mydsp_data_t *mydata = (mydsp_data_t *)dsp_state->plugindata;
result = dsp_state->functions->getblocksize(dsp_state, &blocksize);
ERRCHECK(result);
*data = (void *)mydata;
*length = blocksize * 2 * sizeof(float);
return FMOD_OK;
}
return FMOD_ERR_INVALID_PARAM;
}
/*
Callback called when DSP::setParameterFloat is called. This accepts a floating point 0 to 1 volume value, and stores it.
We have set up 'parameter 1' to be the volume parameter, so it checks to make sure the passed in index is 1, and nothing else.
*/
FMOD_RESULT F_CALLBACK myDSPSetParameterFloatCallback(FMOD_DSP_STATE *dsp_state, int index, float value)
{
if (index == 1)
{
mydsp_data_t *mydata = (mydsp_data_t *)dsp_state->plugindata;
mydata->volume_linear = value;
return FMOD_OK;
}
return FMOD_ERR_INVALID_PARAM;
}
/*
Callback called when DSP::getParameterFloat is called. This returns a floating point 0 to 1 volume value.
We have set up 'parameter 1' to be the volume parameter, so it checks to make sure the passed in index is 1, and nothing else.
An alternate way of displaying the data is provided, as a string, so the main app can use it.
*/
FMOD_RESULT F_CALLBACK myDSPGetParameterFloatCallback(FMOD_DSP_STATE *dsp_state, int index, float *value, char *valstr)
{
if (index == 1)
{
mydsp_data_t *mydata = (mydsp_data_t *)dsp_state->plugindata;
*value = mydata->volume_linear;
if (valstr)
{
sprintf(valstr, "%d", (int)((*value * 100.0f)+0.5f));
}
return FMOD_OK;
}
return FMOD_ERR_INVALID_PARAM;
}
int FMOD_Main()
{
FMOD::System *system;
FMOD::Sound *sound;
FMOD::Channel *channel;
FMOD::DSP *mydsp;
FMOD::ChannelGroup *mastergroup;
FMOD_RESULT result;
unsigned int version;
void *extradriverdata = 0;
Common_Init(&extradriverdata);
/*
Create a System object and initialize.
*/
result = FMOD::System_Create(&system);
ERRCHECK(result);
result = system->getVersion(&version);
ERRCHECK(result);
if (version < FMOD_VERSION)
{
Common_Fatal("FMOD lib version %08x doesn't match header version %08x", version, FMOD_VERSION);
}
result = system->init(32, FMOD_INIT_NORMAL, extradriverdata);
ERRCHECK(result);
result = system->createSound(Common_MediaPath("stereo.ogg"), FMOD_LOOP_NORMAL, 0, &sound);
ERRCHECK(result);
result = system->playSound(sound, 0, false, &channel);
ERRCHECK(result);
/*
Create the DSP effect.
*/
{
FMOD_DSP_DESCRIPTION dspdesc;
memset(&dspdesc, 0, sizeof(dspdesc));
FMOD_DSP_PARAMETER_DESC wavedata_desc;
FMOD_DSP_PARAMETER_DESC volume_desc;
FMOD_DSP_PARAMETER_DESC *paramdesc[2] =
{
&wavedata_desc,
&volume_desc
};
FMOD_DSP_INIT_PARAMDESC_DATA(wavedata_desc, "wave data", "", "wave data", FMOD_DSP_PARAMETER_DATA_TYPE_USER);
FMOD_DSP_INIT_PARAMDESC_FLOAT(volume_desc, "volume", "%", "linear volume in percent", 0, 1, 1);
strncpy(dspdesc.name, "My first DSP unit", sizeof(dspdesc.name));
dspdesc.version = 0x00010000;
dspdesc.numinputbuffers = 1;
dspdesc.numoutputbuffers = 1;
dspdesc.read = myDSPCallback;
dspdesc.create = myDSPCreateCallback;
dspdesc.release = myDSPReleaseCallback;
dspdesc.getparameterdata = myDSPGetParameterDataCallback;
dspdesc.setparameterfloat = myDSPSetParameterFloatCallback;
dspdesc.getparameterfloat = myDSPGetParameterFloatCallback;
dspdesc.numparameters = 2;
dspdesc.paramdesc = paramdesc;
result = system->createDSP(&dspdesc, &mydsp);
ERRCHECK(result);
}
result = system->getMasterChannelGroup(&mastergroup);
ERRCHECK(result);
result = mastergroup->addDSP(0, mydsp);
ERRCHECK(result);
/*
Main loop.
*/
do
{
bool bypass;
Common_Update();
result = mydsp->getBypass(&bypass);
ERRCHECK(result);
if (Common_BtnPress(BTN_ACTION1))
{
bypass = !bypass;
result = mydsp->setBypass(bypass);
ERRCHECK(result);
}
if (Common_BtnPress(BTN_ACTION2))
{
float vol;
result = mydsp->getParameterFloat(1, &vol, 0, 0);
ERRCHECK(result);
if (vol > 0.0f)
{
vol -= 0.1f;
}
result = mydsp->setParameterFloat(1, vol);
ERRCHECK(result);
}
if (Common_BtnPress(BTN_ACTION3))
{
float vol;
result = mydsp->getParameterFloat(1, &vol, 0, 0);
ERRCHECK(result);
if (vol < 1.0f)
{
vol += 0.1f;
}
result = mydsp->setParameterFloat(1, vol);
ERRCHECK(result);
}
result = system->update();
ERRCHECK(result);
{
char volstr[32] = { 0 };
FMOD_DSP_PARAMETER_DESC *desc;
mydsp_data_t *data;
result = mydsp->getParameterInfo(1, &desc);
ERRCHECK(result);
result = mydsp->getParameterFloat(1, 0, volstr, 32);
ERRCHECK(result);
result = mydsp->getParameterData(0, (void **)&data, 0, 0, 0);
ERRCHECK(result);
Common_Draw("==================================================");
Common_Draw("Custom DSP Example.");
Common_Draw("Copyright (c) Firelight Technologies 2004-2017.");
Common_Draw("==================================================");
Common_Draw("");
Common_Draw("Press %s to toggle filter bypass", Common_BtnStr(BTN_ACTION1));
Common_Draw("Press %s to decrease volume 10%", Common_BtnStr(BTN_ACTION2));
Common_Draw("Press %s to increase volume 10%", Common_BtnStr(BTN_ACTION3));
Common_Draw("Press %s to quit", Common_BtnStr(BTN_QUIT));
Common_Draw("");
Common_Draw("Filter is %s", bypass ? "inactive" : "active");
Common_Draw("Volume is %s%s", volstr, desc->label);
if (data->channels)
{
char display[80] = { 0 };
int channel;
for (channel = 0; channel < data->channels; channel++)
{
int count,level;
float max = 0;
for (count = 0; count < data->length_samples; count++)
{
if (fabs(data->buffer[(count * data->channels) + channel]) > max)
{
max = fabs(data->buffer[(count * data->channels) + channel]);
}
}
level = max * 40.0f;
sprintf(display, "%2d ", channel);
for (count = 0; count < level; count++) display[count + 3] = '=';
Common_Draw(display);
}
}
}
Common_Sleep(50);
} while (!Common_BtnPress(BTN_QUIT));
/*
Shut down
*/
result = sound->release();
ERRCHECK(result);
result = mastergroup->removeDSP(mydsp);
ERRCHECK(result);
result = mydsp->release();
ERRCHECK(result);
result = system->close();
ERRCHECK(result);
result = system->release();
ERRCHECK(result);
Common_Close();
return 0;
}Thank you Brett,
I took some time to digest all of this and I’ve never been closer to visualizing wave data with FMOD 5. There is one thing that remains ambiguous though. The length_samples return 1024 in my case. If my understanding is correct, there is no way to change this, or at least, it is not recommended right?
I’m trying to make the wave data to fit in a 512 float array. How would you suggest to proceed while keeping the most accurate result?
Also, since it is one dimension array, I must down mix all channels into a one. Would this be solved by averaging all channels?
Then again, if my understanding is correct, the out buffer channel is a ring buffer, should the 512 float array be reset before every updated?
There is a way to change it, but if you want to affect your audio, you wouldn’t, and you don’t need to change it.
If you want to go smaller, 512 actually is reasonable to change with setDSPBufferSize. I just wouldn’t go bigger than 1024 or smaller than 256. If you halve the block size, double the numbuffers. if you quarter the size, quadruple the numbuffers. This keeps the total buffer size the same at least.
My new demo pasted above, uses max(). If you want 1 value out of a stereo signal, It would probably make more sense to use max, not average. If one side is silent and the other side is full volume, would you rather see a level that represents the max, or a level that is only half way?
For arbitrary sizes, you should create a ringbuffer yourself (ie data->buffer), and keep a cursor position of where the last 1024 block was written to. then you can read as much or as little as you like from the cursor position backwards. This is beyond the scope of something like this example that I have pasted above (i have edited it so that it can be included in the next public API release.
That is very handy. Thank you Brett.
dsp_state->functions seems to translate to dsp_state->callbacks but nonetheless, we’re getting wave data! We’re almost there as I have a few more questions.
Since the out buffer doesn’t need to be modified, (or does it?)
memcpy(dsp_state->plugindata, in_buffer, in_length * in_channels * sizeof(Float));
Having only the wave data in mind, would this be copying the buffer correctly or I’m missing something?
The wave data value count should match the FMOD_DSP_FFT_SPECTRUMDATA buffer in this case. How should we set the equivalent of FMOD_DSP_FFT_WINDOWSIZE to the wave data buffer?
Common_Draw("buffer = … seems to be cutting corners. For the sake of having a complete example and once again, if it is not too much too ask. How about adding the drawing of all values including every channels to the code?
The wave data values seems to be very small, is there a common way to amplify the result with FMOD 5? Could the values be scaled linearly?
The wave data values seems to be very edgy, is there a common way to smooth the result with FMOD 5?