Offline rendering of elements - recreating an audio app

FMOD Engine
,
1

Hello! We are working on a game which features several “audio applications”. In our game, you open up audio files and work on them.

So we were wondering how to properly display sound data without having to rely on realtime playback to get DSP data. Basically, we would need to re-create some parts of a standard audio editing program, with the audio data being presented faster-than-realtime.

Any suggestions on how to tackle this is highly appreciated!

Filippo

2

Hi,

You are able to extract PCM data without playing the sound: FMOD Engine | Advanced Core Api Topics - Extracting Pcm Data From A Sound would this be a valid option?

Is this a similar issue as you were experiencing before: Visualizing both classic Loudness x Time and Spectrogram?

3

Hello Connor, hhank you for your answer!

Yes, it is a similar issue as we were experiencing before. Since my last post, we have explored our options in this regard. As far as we could see, the approach you posted requires audio files to be present outside of FMOD.

Since we handle all our game’s audio through FMOD, this does not really work for us. We haven’t found a way to read data in the same way from an FMOD event, or through the banks. We tried playing around with DSPs extensively.
However, they require the event to run in order to listen to data, and also cannot do it faster than realtime. We’d need the entire audio clip/event data as fast as possible. We have read about custom DSP plugins. Would that be able to bypass these limitations then? We used timeline markers and event user properties to pass custom per-event-data in our current prototype, but that made our workflow much more error prone and tedious.

If you know a way to query advanced info of events and their components (sub events, clips, effects, parameters, etc.) from outside FMOD, that’d be very useful to us. Thank you in advance!

4

Hi Connor,

as an additional way to explain what we would need, our coder made a little “interactive” explainer quickly: https://gemini.google.com/share/698be75ec6fb

We’re basically trying to (partly) recreate an audio editor :sweat_smile:

Let us know if you need any more details, happy to answer in any way we can!

Filippo

5

Dear Connor,

just checking if you could give us some pointers as to how to solve our conundrum :smiley: Let me know if you need any more input from my side. Thanks!

Cheers from Munich,

Filippo

6

Sorry about the delay, Connor has been away at conferences and it looks like this case didn’t get reassigned for some reason.
There is no way to combine realtime and non-realtime DSP processing in FMOD because that’s essentially a system-level setting (FMOD.OUTPUTTYPE), but if you store your loose audio assets in a folder you could use a Programmer Instrument to play them back, and manipulate the sample data inside the CREATE_PROGRAMMER_SOUND callback. I haven’t tested this, but based off of the Unity Programmer Sound Example you could do something like:

case FMOD.Studio.EVENT_CALLBACK_TYPE.CREATE_PROGRAMMER_SOUND:
{
    FMOD.MODE soundMode = FMOD.MODE.LOOP_NORMAL | FMOD.MODE.CREATECOMPRESSEDSAMPLE;
    var parameter = (FMOD.Studio.PROGRAMMER_SOUND_PROPERTIES)Marshal.PtrToStructure(parameterPtr, typeof(FMOD.Studio.PROGRAMMER_SOUND_PROPERTIES));
    FMOD.Sound dialogueSound;
    var soundResult = FMODUnity.RuntimeManager.CoreSystem.createSound(Application.streamingAssetsPath + "/" + key, soundMode, out dialogueSound);
    if (soundResult == FMOD.RESULT.OK)
    {
        dialogueSound.getLength(out uint soundLengthBytes, TIMEUNIT.RAWBYTES);

        var res = dialogueSound.@lock(0, soundLengthBytes, out IntPtr ptr1, out IntPtr ptr2, out uint lenBytes1, out uint lenBytes2);
        if (res != FMOD.RESULT.OK) Debug.LogError(res);

        /*
         * Though exinfo.format is float, data retrieved from Sound::lock is in bytes,
         * therefore we only copy (len1+len2)/sizeof(float) full float values across
         */
        int lenFloats1 = (int)(lenBytes1 / sizeof(float));
        int lenFloats2 = (int)(lenBytes2 / sizeof(float));
        int totalFloatsRead = lenFloats1 + lenFloats2;
        float[] tmpBufferFloats = new float[totalFloatsRead];

        for(int i = 0; i < totalFloatsRead; i++)
        {
            // Modify sound data, in this case just half volume
            tmpBufferFloats[i] *= 0.5f;
        }

        // Copy data back to pointers
        if (lenBytes1 > 0)
        {
            Marshal.Copy(tmpBufferFloats, 0, ptr1, lenFloats1);
        }
        if (lenBytes2 > 0)
        {
            Marshal.Copy(tmpBufferFloats, lenFloats1, ptr2, lenFloats2);
        }

        // Push pointers back to FMOD Sound
        res = mSound.unlock(ptr1, ptr2, lenBytes1, lenBytes2);

        parameter.sound = dialogueSound.handle;
        parameter.subsoundIndex = -1;
        Marshal.StructureToPtr(parameter, parameterPtr, false);
    }
}