Capturing Unity game’s FMOD output (5.1 / Busses)

Hi,
I am about to start working on a game project in Unity using FMOD. Some scenes will be captured as video and later used for a linear 5.1 mix. I am wondering what the most elegant ways are to capture FMODs audio output in sync (can be a separate WAV output file) with Unity’s video on windows.

Specifically in 5.1 and in Stereo.

And related: how can I capture my FMOD audio busses to separate WAV files during the same Unity run(In my case those groups are just two: ambiences and soundeffects.

If we need custom coding for that I’d be grateful for any pointers how to potentially approach this, if there some way to do it directly via Unity or FMOD I’d love hear about it.

Or maybe there is a third party tool or somebody already coded a solution to my problem which I could use a a starting point?

Thanks a lot!

Hi,

FMOD has the FMOD_OUTPUTTYPE_WAVWRITER output mode that allows the system to write the final mixer output to a .wav file. Note that using WAVWRITER means that the system will solely output to file; it won’t output to a device. To accomplish this with the Unity Integration, you’ll have to modify FMODUnity.RuntimeManager as follows:

  • Add the following as class members:
byte[] wavPath;
GCHandle wavHandle;
IntPtr wavPtr;
  • Modify the output of the coreSystem object on line 301:
result = coreSystem.setOutput(FMOD.OUTPUTTYPE.WAVWRITER);
  • Replace the studioSystem.initialize() call on line 347 with the following:
using (FMOD.StringHelper.ThreadSafeEncoding encoder = FMOD.StringHelper.GetFreeHelper())
{
    wavPath = encoder.byteFromStringUTF8("C:\\Your\\Path\\Here\\wavFile.wav");
    wavHandle = GCHandle.Alloc(wavPath, GCHandleType.Pinned);
    wavPtr = wavHandle.AddrOfPinnedObject();
    result = studioSystem.initialize(virtualChannels, studioInitFlags, FMOD.INITFLAGS.NORMAL, wavPtr);
}
  • Free the GCHandle object wavHandle in RuntimeManager.OnDestroy() after ReleaseStudioSystem() is called:
private void OnDestroy()
{
    coreSystem.setCallback(null, 0);
    ReleaseStudioSystem();

    wavHandle.Free();
    //...
}

You will also need to make the static class StringHelper in fmod.cs at line 3884 public.

However, this won’t allow you to separately capture audio for different Buses unless you do multiple recordings with the Buses muted/solo’d. To do this, you’ll need to go a little more in-depth - I would suggest modifying our Unity DSP Capture script example to collect the audio data, adding a wav header to the data, and writing it to a .wav file. The DSP Capture example specifically grabs the Master ChannelGroup’s output, but you can modify it to grab any Bus’ core ChannelGroup instead, which you can get with Studio::Bus::getChannelGroup. As for writing to a wav file with the appropriate header, here’s a somewhat old post detailing how to record audio input and write it to a wav file. The post uses in C++, and specifically pertains to writing audio input to file, but you should be able to adapt it to C# and your specific use-case.

If you have any additional questions, feel free to ask.

Hi @Leah_FMOD ! I’m a teammate of @davidkamp1 and your comment was very helpful. I tested the WAVWriter and that indeed records a .wav file of the whole master and so on.

However, due to wanting to record various buses and other specs, I moved onto trying to accomplish what you mentioned: DSP Capture script example with the .wav header and writing to a .wav file. (for now using only the Master ChannelGroup until that works, will test a different bus later).

Now, I am a gameplay programmer and I’m completely out of my depth once it comes to structured data like this. The C++ to C# isn’t a problem, but the part that I don’t understand at all is how I should create the file out of the data from the DSP. As in, I don’t know what to connect where, how to feed the float array mDataBuffer from the DSP example to a file or how to create a header based on that data.

I have tried using this and both solutions provided here for reference, but every time the examples are about recording from microphone with the functions that come with an initialized system and it’s throwing me off. I don’t know how to implement this.

Could you please help me out a little? Thank you!

In case it’s relevant, our usecase is to save the FMOD sounds of a certain amount of gameplay into .wav files, so we can replay that segment later together synced with video that we record with a different tool, not to record all gameplay sounds since the Scene starts.

Hi @Leah_FMOD - Thanks a lot for the initial pointers - Would you mind helping us out a bit more? We are still trying to capture two mixer groups seperately. The three mixer groups we try to capture are named in FMOD Studio as follows:
Environments
Characters
Obstacles

@Leah_FMOD Do you think you can create a working example for us by modifying the DSP capture example you mentioned so it works with the three above busses and writes to WAV? That would be amazing, since @xantomen seems to run into issues… and I am sure having such a working example is helpful for other FMOD users too! Thanks so much for your support so far.

Hi @xantomen and @davidkamp1,

The fundamental approach to this issue can be split into capturing the audio, and writing the audio to a WAV.

Capturing is a bit simpler. You’d essentialy just want to attach an instance of the capture DSP to each of the channel groups underlying each bus (alternatively, place a single capture DSP on a bus that the 3 buses route into) - you should then be able to sum the sample from each to get the combined signal.

Writing to WAV is a little more difficult, depending on your approach and needs. Based on what you’ve described, I think the most suitable approach would be to handle writing to WAV yourself instead of creating an additional FMOD system to use the WAVWRITER outout. A naive way to do this might be to simply keep all of the captured audio in memory, and then write it to disk when desired. A slightly improved version would be to write the captured buffer on the fly, which obviously saves on memory.

Regardless, I’ll take a look at creating a simple example script on my end soon and get back to you.

1 Like

Hi @Leah_FMOD . Thank you for your reply!

The theory of the approach is all clear to me, it’s just once it comes down to the code I’m really lost. Especially on how to feed the captured buffer data into the .wav file and the header.

A code sample would really help me out, thanks for offering that!

Thanks for your patience @xantomen and @davidkamp1.

The following DSPCaptureAndWrite script should serve as a straightforward example of how to accomplish what you want. It adds a capture DSP to a specified Bus’s underlying ChannelGroup (similar to our Unity DSP Capture example), and captures the audio data in memory. Then, on “R” being input, it will write the captured audio data to WAV file a specified file path and clear the audio data stored in memory.

DSPCaptureAndWrite.cs
using System;
using UnityEngine;
using System.Runtime.InteropServices;
using System.Collections.Generic;
using System.IO;

class DSPCaptureAndWrite : MonoBehaviour
{
    [SerializeField]
    private string mFilePath;
    [SerializeField]
    private string mBusPath;

    private FMOD.DSP_READ_CALLBACK mReadCallback;
    private GCHandle mObjHandle;
    private FMOD.DSP_DESCRIPTION mDSPDescription;

    private FMOD.DSP mDSP;
    private FMOD.ChannelGroup mCG;
    private List<float> mAudioData;

    private int mSampleRate;
    private int mNumChannels;
    private int mBitDepth = 32; // Assumes data will be 32 bit PCM Float, which it will automatically be in the FMOD Mixer graph

    private bool mDSPsCreated;
    private bool mRecording;
    private FileStream fs;
    private BinaryWriter bw;


    [AOT.MonoPInvokeCallback(typeof(FMOD.DSP_READ_CALLBACK))]
    static FMOD.RESULT CaptureDSPReadCallback(ref FMOD.DSP_STATE dsp_state, IntPtr inbuffer, IntPtr outbuffer, uint length, int inchannels, ref int outchannels)
    {
        // Copy the input buffer to an intermediate buffer
        int lengthElements = (int)length * inchannels;
        float[] data = new float[lengthElements];
        Marshal.Copy(inbuffer, data, 0, lengthElements);

        // Get instance of DSPCaptureAndWrite from user data assigned to DSP
        FMOD.DSP_STATE_FUNCTIONS functions = (FMOD.DSP_STATE_FUNCTIONS)Marshal.PtrToStructure(dsp_state.functions, typeof(FMOD.DSP_STATE_FUNCTIONS));
        IntPtr userData;
        functions.getuserdata(ref dsp_state, out userData);
        if (userData != IntPtr.Zero)
        {
            GCHandle objHandle = GCHandle.FromIntPtr(userData);
            DSPCaptureAndWrite obj = objHandle.Target as DSPCaptureAndWrite;

            // If currently recording, add buffer contents to recorded audio data
            if (obj.mRecording)
            {
                obj.mNumChannels = inchannels;
                obj.mAudioData.AddRange(data);
            }
        }

        // Copy the input buffer to the output buffer so we can still hear it
        Marshal.Copy(data, 0, outbuffer, lengthElements);

        return FMOD.RESULT.OK;
    }

    void Start()
    {
        // Assign the DSP capture callback to a member variable to avoid garbage collection
        mReadCallback = CaptureDSPReadCallback;

        // Get DSP/Audio info and initialize list used to to store capture audio
        uint bufferLength;
        FMODUnity.RuntimeManager.CoreSystem.getDSPBufferSize(out _, out _);
        FMODUnity.RuntimeManager.CoreSystem.getSoftwareFormat(out mSampleRate, out _, out _);
        mAudioData = new();

        // Get a handle to this object to pass into the callback
        mObjHandle = GCHandle.Alloc(this, GCHandleType.Pinned);
        if (mObjHandle != null)
        {
            // Define a basic DSP that receives a callback each mix to capture audio
            mDSPDescription = new FMOD.DSP_DESCRIPTION();
            mDSPDescription.numinputbuffers = 1;
            mDSPDescription.numoutputbuffers = 1;
            mDSPDescription.read = mReadCallback;
            mDSPDescription.userdata = GCHandle.ToIntPtr(mObjHandle);
        }
        else
        {
            Debug.LogWarningFormat("FMOD: Unable to create a GCHandle: mObjHandle");
        }

        // Don't start recording until DSPs have been created
        mRecording = false;
        mDSPsCreated = false;

    }

    void Update()
    {
        // Bus' underlying ChannelGroup needs to be active before DSPs can be added
        if (!mDSPsCreated)
        {
            FMOD.Studio.Bus bus = FMODUnity.RuntimeManager.GetBus(mBusPath);
            mRecording = AddDSP(bus);
            mDSPsCreated = mRecording;
            if (mRecording) Debug.Log("FMOD: Started capturing audio data");
        }

        if (mDSPsCreated && mRecording)
        {
            // When recording, press R to stop writing to file
            if (Input.GetKey(KeyCode.R))
            {
                Debug.Log("FMOD: Stopped capturing audio data, writing audio to file at " + mFilePath);
                mRecording = false;
                fs = File.Create(mFilePath);
                bw = new BinaryWriter(fs);
                WriteWavHeader(mAudioData.Count);
                byte[] bytes = new byte[mAudioData.Count * 4];
                Buffer.BlockCopy(mAudioData.ToArray(), 0, bytes, 0, bytes.Length);
                fs.Write(bytes);
                fs.Close();
                bw.Close();
                mAudioData.Clear();
            }

            
        }
    }

    bool AddDSP(FMOD.Studio.Bus bus)
    {
        FMOD.DSP captureDSP = new FMOD.DSP();
        FMOD.ChannelGroup cg;
        if (bus.getChannelGroup(out cg) == FMOD.RESULT.OK)
        {
            if (FMODUnity.RuntimeManager.CoreSystem.createDSP(ref mDSPDescription, out captureDSP) == FMOD.RESULT.OK)
            {
                if (cg.addDSP(0, captureDSP) != FMOD.RESULT.OK)
                {
                    Debug.LogWarningFormat("FMOD: Unable to add DSP to the bus' channel group");
                }
                else
                {
                    mDSP = captureDSP;
                    mCG = cg;
                    return true;
                }
            }
            else
            {
                Debug.LogWarningFormat("FMOD: Unable to create a DSP");
            }
        }
        else
        {
            Debug.LogWarningFormat("FMOD: Unable to get bus' channel group");
        }
        captureDSP.release();
        return false;
    }

    void RemoveDSP()
    {
        // If the DSP is valid, remove it from the ChannelGroup and release
        if (mDSP.hasHandle())
        {
            mCG.removeDSP(mDSP);
            mDSP.release();
        }
    }

    void OnDestroy()
    {
        if (mObjHandle != null)
        {
            RemoveDSP();
            mObjHandle.Free();
        }
    }

    void WriteWavHeader(int length)
    {
        bw.Seek(0, SeekOrigin.Begin);

        bw.Write(System.Text.Encoding.ASCII.GetBytes("RIFF"));      //RIFF                          4 bytes chars
        bw.Write(32 + length * 4 - 8);                              //File Size (after this chunk)  4 bytes int     (32 for rest of header + wave data)

        bw.Write(System.Text.Encoding.ASCII.GetBytes("WAVEfmt "));  //WAVEfmt                       8 bytes chars
        bw.Write(16);                                               //Length of above fmt data      4 bytes int
        bw.Write((short)3);                                         //Format 1 is PCM               2 bytes short
        bw.Write((short)mNumChannels);                              //Number of Channels            2 bytes short
        bw.Write(mSampleRate);                                      //Sample Rate                   4 bytes int
        bw.Write(mSampleRate * mBitDepth / 8 * mNumChannels);       //                              4 bytes int
        bw.Write((short)(mBitDepth / 8 * mNumChannels));            //                              2 bytes short
        bw.Write((short)mBitDepth);                                 //Bits per sample               2 bytes short

        bw.Write(System.Text.Encoding.ASCII.GetBytes("data"));      //data                          4 bytes chars
        bw.Write(length * 4);                                       //Size of data section          4 bytes int
    }
}

Note that an instance of this class will only capture audio from a single bus. If you want to capture each of your three buses individually, you can just use three instances of this class and write the audio data to seperate WAV files.

If you want to capture and combine the audio from each of your buses, the script will require a little modification, but it should be as simple as placing an instance of the capture DSP on each bus, storing the captured audio data in three separate lists, and then summing each element of the three lists together before writing to file.

If you run into any issues using the script, please feel free to let me know.