Adding DSP Causes WASAPI Starvation

FMOD Engine
#1

I’m writing a system to forward all of the audio coming out of FMOD to FFMPEG, for video recording. Right now I have achieved this with a custom DSP attached to the tail of the master channel group. Functionally, everything works, except that after this DSP is added, at the end of gameplay I get a starvation warning and an audible ‘pop’, when the FMOD core system shuts down.

Here are the minimal reproduction steps for me:

  1. At the start of the game, add a custom DSP, with an empty C# callback.
  2. At frame 5, remove and release the custom DSP.
  3. At frame 5000, the game ends, and I hear a pop, and get a starvation warning.

Is there some reason that having an empty C# DSP attached to the master group might cause a delay? I can’t imagine why I see this issue at the end of the game, even though the DSP is long gone (removed on frame 5). I’ve confirmed it’s removed properly using the FMOD Core Profiler.

Here’s my code. Is there a better way to read audio out of the master bus? Ideally, I would like my reading to be asynchronous, but as far as I can tell, there’s no way to achieve that?

var dspDescription = new DSP_DESCRIPTION
        {
            version = 0x00010000,
            name = dspName,
            numinputbuffers = 1,
            numoutputbuffers = 1,
            read = DspReadCallback,
            numparameters = 0
        };

// Get the master group and read the channel settings.
FMOD.System system = RuntimeManager.CoreSystem;
CheckError(system.getMasterChannelGroup(out ChannelGroup masterGroup));
CheckError(masterGroup.getDSP(CHANNELCONTROL_DSP_INDEX.TAIL, out DSP masterDspTail));
CheckError(masterDspTail.getChannelFormat(out CHANNELMASK channelMask, out int numChannels, out SPEAKERMODE sourceSpeakerMode));

// Create a new DSP with the format of the existing master group.
CheckError(system.createDSP(ref dspDescription, out dsp));
CheckError(dsp.setChannelFormat(channelMask, numChannels, sourceSpeakerMode));
CheckError(masterGroup.addDSP(CHANNELCONTROL_DSP_INDEX.TAIL, dsp));

// ... 

// Copy audio into buffer. Issue appears, even if this function is empty.
private RESULT DspReadCallback(ref DSP_STATE dsp_state, IntPtr inbuffer, IntPtr outbuffer, uint samples, int inchannels, ref int outchannels)
#2

Using a custom DSP is the best way to get the audio stream out of the game. If you need async access you would need to buffer the audio in the read callback and consume it as needed from another thread.

When you say an empty read callback, you still need to pass audio from inbuffer to outbuffer or everything will be silenced.

It’s interesting you mention the DSP has been disconnected and the pop happens a long time later, I’m not sure how the DSP is related to the pop but you mention it only happens if the DSP has a read callback? Do you still get the pop after the DSP is released (not just disconnected?). What about creating one of the built in DSP, do you have the same issue using them or is it only custom DSP?

#3

Thanks for the reply!

Right, when I mean empty, I mean a trivial copy buffer dsp. In C#:

// Pass the input through to the output, so we can still hear it.
unsafe
{
     Buffer.MemoryCopy(inBuffer.ToPointer(), outBuffer.ToPointer(),
          blockSizeBytes,
          blockSizeBytes);
}

I have also tried a truly empty DSP, which doesn’t produce audio, but oddly still throws the buffer starvation warning!

For your questions: Yes, it only seems to happen with a custom DSP, with a C# read callback. In all of these tests, I have been releasing the DSP after removing it.

I just tried your suggestion of a built-in DSP:

CheckError(system.createDSPByType(DSP_TYPE.ECHO, out dsp));
CheckError(masterGroup.addDSP(CHANNELCONTROL_DSP_INDEX.TAIL, dsp));

This works just fine! No starvation. So something is specifically wrong with using a custom DSP.

I can’t help but wonder if this is a problem with calling into the .NET runtime from C++. Is there weird locking behavior, or some sort of extra setup that is only triggered when a C# callback is passed into the native layer?

Even though I’m removing the DSP earlier, this pop is happening when the FMOD System shuts down. What if adding a C# callback causes the core system to hold onto a handle into the .NET runtime? That might cause a hang, if FMOD is waiting on the .NET runtime during shutdown.

#4

I haven’t been able to reproduce the problem you are describing.
Could you provide a simple Unity project that demonstrates it happening?

#5

Was this issue fixed? I have literally the exact same bug as described by OP – a custom C# DSP on the master group causes a pop when the studio system shuts down and a buffer starvation warning (which subsequently causes Unity to freeze upon play again).

I hope there’s an easy resolution to this bug!

#6

Unfortunately I have not yet solved it. It isn’t a blocking issue for us, and our project is rather challenging to distill down into a simple project to send to Matthew.

Maybe your project is smaller?

#7

If anyone is able to provide a simple demo that shows this happening it will go a long way towards us being able to solve it.

#8

So I made a simple demo that uses the same code that breaks my main project but the bug actually does not happen. I would guess this is because my main project has much more complex behavior overall and uses more CPU to start and stop a scene, and as far as I can tell the bug happens because FMOD gets a buffer starvation upon closing the scene and that seems to be related to the C# DSP in some way.

One thing I did notice in the demo project was that if you set the buffer to be too small to even play an event, you get a similar behavior where stopping and then restarting the scene causes Unity to hang permanently. That seems to be a different problem but could be a similar bug if that is unintended behavior?

Here is a link to the demo project I made. Again this doesn’t replicate the bug, but it does replicate the exact code that causes the main project to bug. If you all have an easy way to add more complexity to the scene (or FMOD) so there is more CPU usage that might cause it to happen. You can also test setting the buffer size to a tiny amount and seeing the hang it causes.

If it is ultimately needed, I could try somehow duplicating the exact scenario in my main project, but that is a much more complex undertaking so I’m hoping this can suffice. Thanks for your help!

#9

Do you have the problem if you run your game as a standalone build using IL2CPP?

#10

When building a standalone to OSX using IL2CPP I get the following error:

NotSupportedException: IL2CPP does not support marshaling delegates that point to instance methods to native code. The method we're attempting to marshal is: FMOD_CustomDSP::ReadAudioData > FMOD.System.FMOD5_System_RegisterDSP (System.IntPtr system, FMOD.DSP_DESCRIPTION& description, System.UInt32& handle)....

Since my demo project uses the same code, I think this error should be reproducible if you build it using IL2CPP.

Since this exception happens, I am not sure if the DSP bug that happens in the editor reproduces in the build. But the primary bug in the editor is that it fails to run the project a 2nd time after the FMOD ring buffer starves on stopping playmode once the DSP has loaded and ran, so I’m not entirely sure how that would reproduce in a build since you aren’t restarting it like the editor does.

#11

Even with your project I am unable to reproduce the small buffer size == hang scenario. I can see you are hand coding the buffer size to 256 x 4, this works fine in my tests, going lower seems okay too.

Additionally building standalone windows with your project produces no errors when switched to IL2CPP.

Is this happening on Windows like the OP describes? or are your issues Mac only?
I should be able to test this on Mac next week if that’s the case for you.

#12

Interesting – setting the buffer size to 128 x 4 causes a hang on the consequent editor Play every time I test it. I just tested this in my test project and it reproduces consistently for me.

Unfortunately I only have a Mac testing environment right now so I am not sure if it reproduces to Windows. Here is my environment: macOS Catalina 10.15.7, Unity 2019.4.11f1 and FMOD 2.01.05.

I’ll be interested to know your Mac testing results. Thanks for you help.

#13

Testing on Mac I can see setting the buffer to 128x4 isn’t workable. You should be seeing lots of warnings in the Unity console explaining what’s going on. For my Mac at least, a minimum of 1024 samples is required to service the operating system requests. 256x4 works, just, 128x4 gives a warning saying the buffer size is potentially too small, then lots of warnings, one for each mixer update that a request was made larger than FMOD can provide.

#14

Yes, you are correct that the 128x4 is not workable, and I do get the same warnings that you see when setting the buffer size to be too small.

I was setting the buffer size to a small amount in an attempt to reproduce the bug that I see in my main project. I don’t actually use 128x4 in my main project (use is the default 512x4), but it was an idea to see if the hang error could be reproduced by setting the buffer size to a smaller amount, since it me it seemed related to a buffer starvation.

Would it be helpful to send my main project to you? I can build the scene that gets the DSP starvation issue only (although it’ll still be about 512 MB). Since my test project doesn’t reproduce the exact bug, that might be the easiest way to show you since my buffer-size hacking doesn’t actually reproduce it and introduces other issues.

#15

Yes, if you’re able to send me something that reproduces the bug that will make things much easier to diagnose and fix. If you’d prefer to keep that data private you can upload using your FMOD account here: https://fmod.com/profile#uploads

#16

Strangely enough, while I was creating a cut down version of my project to reproduce the bug, it stopped happening, in both my main and the cut down project. I do not believe I updated Unity or FMOD in that timeframe but perhaps a minor update to Catalina fixed it? I am really not sure what happened, it used to reproduce nearly every time…

If the bug begins happening again (perhaps it was some OS/System Memory state) I will send through all I have to you but for now consider this topic closed on my end. Thanks for your help!

#17

Bumping this thread again, turns out the bug never went away, it instead became a freeze on compilation that I thought was due to another system (asmdef files). I have been looking into this one again and I am almost certain that this has to do with a memory leak when using C# for a custom dsp. I have watched the memory balloon over consecutive plays of the Unity editor and I believe the reason why my small test project did not reproduce the bug was due to that it uses barely any memory since it is a test scenario.

Is there an easy way to diagnose memory leaks when using the FMOD Core API? Assuming my logic is correct, the test project I sent earlier in this thread should have the same memory leak, it simply won’t cause the editor to hang because the project doesn’t use enough memory to go over whatever boundary is causing the hang. The code in my main project is almost identical to the test project (one different line where I don’t release the channel group since that seems to break Unity) so I’d think it should be easy to diagnose from the test project.

#18

Thanks for your patience on this issue. It sounds like the freeze could be due to callbacks from C++ being fired during an assembly reload operation. We’re looking into it at the moment, and will let you know what we find.