Dynamic Audio + Low Latency + 3D Spatialization

stephen_tool1 · June 17, 2024, 10:35am

Hey all,

I feel like I am running in circles with my usage of FMOD. I am trying to integrate a custom VoIP service while supporting 3D Spatialization (I want to use Odin by 4players https://www.4players.io/products/voice-chat/)

My first approach was to use a Programmer Sound and just copy the input of the VoIP over to the pcmreadcallback buffer of it. (I have something like this https://www.fmod.com/docs/2.02/unity/examples-programmer-sounds.html, adapted to the Unreal Engine). This worked pretty well, except the latency is really high (~500-800ms) which is unacceptable in voice chat.
So after a while of searching I found out, that the Programmer Sound seems to add a latency to cover up potential loading times of sounds. I could not figure out how to change that artificial latency so I looked for another approach.

So now I am using a custom DSP that I define during runtime and add to the master channel group for testing purposes. All fine - the latency is gone but now I am not sure how to add Spatialization. Again a lot of searching in the documentation and I came across the DSP Pan Plugin that I add to the same channel group after my custom DSP. (https://www.fmod.com/docs/2.01/api/effects-reference.html#pan) But no matter how I configure that Plugin, it just plays the sound as a 2D sound without any spatialization or volume attenuation. Is there any guide on this? Or anyone who can give me the correct directions? I cannot find anything in the FMOD documentation that seems to work in this use case.

Or: Is there any chance, that the artificial delay of the programmer sound might be configurable?

Or: Is there a third approach to passing the VoIP data to FMOD that has low latency and allows for spatialization?

Thanks in advance and have good one!

Connor_FMOD · June 19, 2024, 5:03am

Hi,

Thank you for all the information.

What version of the FMOD integration are you using?

Would it be possible to have a look at both of your implementations to see if there is any way I can help?

A user has implemented something similar here: Vivox to Unity OnAudioFilterRead to FMOD programmer sound. Stutters/crackling - #15 by dougmolina. It is using C#, however, it may have some useful insights.

Unfortunately, I think you have touched on both methods.

stephen_tool1 · June 19, 2024, 4:15pm

Hey Connor,

thanks a lot for the fast response!

I will have a look at the post you provided and try if I can solve the latency issue with my own ring buffer.

Meanwhile here is my current implementation. This is all placed inside a class derived from USceneComponent:

void UOdinFmodAdapter::BeginPlay()
{
	FMOD::Studio::System* System = IFMODStudioModule::Get().GetStudioSystem(EFMODSystemContext::Runtime);
	FMOD::System* CoreSystem = nullptr;
	System->getCoreSystem(&CoreSystem);

        // Create Odin DSP
	FMOD_DSP_READ_CALLBACK mReadCallback = OdinDSPReadCallback;
	FMOD::DSP* mOdinDSP = nullptr;

	FMOD_DSP_DESCRIPTION desc = { 0 };
	desc.read = mReadCallback;
	desc.userdata = this;

	FMOD_RESULT res = CoreSystem->createDSP(&desc, &mOdinDSP);

	if (res == FMOD_RESULT::FMOD_OK)
	{
		CoreSystem->getMasterChannelGroup(&group);

		if (group->addDSP(0, mOdinDSP) == FMOD_RESULT::FMOD_OK)
		{
			UE_LOG(LogTemp, Warning, TEXT("Added Odin DSP to channel group"));
		}
	}

        // create Pan DSP
	FMOD_RESULT result = CoreSystem->createDSPByType(FMOD_DSP_TYPE_PAN, &dsp_pan);

	

	if (result == FMOD_RESULT::FMOD_OK)
	{
		CoreSystem->getMasterChannelGroup(&group);

		if (group->addDSP(FMOD_CHANNELCONTROL_DSP_TAIL, dsp_pan) == FMOD_RESULT::FMOD_OK)
		{
			UE_LOG(LogTemp, Warning, TEXT("Added Object Spatializer DSP to channel group"));

			group->setMode(FMOD_3D | FMOD_3D_WORLDRELATIVE | FMOD_3D_LINEARROLLOFF);
		}
	}

	// Set Pan Output to Surround for 3D Spatialization
	dsp_pan->setParameterInt(FMOD_DSP_PAN_MODE, (int)FMOD_DSP_PAN_MODE_SURROUND);

	// Set Pan Mode to full 3D Positional
	dsp_pan->setParameterFloat(FMOD_DSP_PAN_3D_PAN_BLEND, 1.0f);

	UpdateAttenSettings();
	Update3DPosition();
}

void UOdinFmodAdapter::UpdateAttenSettings()
{
	dsp_pan->setParameterInt(FMOD_DSP_PAN_3D_ROLLOFF, (int)RolloffType);
	dsp_pan->setParameterFloat(FMOD_DSP_PAN_3D_MIN_DISTANCE, MinimumDistance);
	dsp_pan->setParameterFloat(FMOD_DSP_PAN_3D_MAX_DISTANCE, MaximumDistance);
	dsp_pan->setParameterInt(FMOD_DSP_PAN_3D_EXTENT_MODE, (int)ExtentMode);
	dsp_pan->setParameterFloat(FMOD_DSP_PAN_3D_SOUND_SIZE, SoundSize);
	dsp_pan->setParameterFloat(FMOD_DSP_PAN_3D_MIN_EXTENT, MinimumExtent);
}

void UOdinFmodAdapter::Update3DPosition()
{
	FMOD_DSP_PARAMETER_3DATTRIBUTES_MULTI attrs = { 0 };

	FMOD_3D_ATTRIBUTES absattr = { 0 };

	absattr.position = ConvertUnrealToFmodVector(this->GetComponentLocation(), 0.01f);
	absattr.velocity = ConvertUnrealToFmodVector(this->GetComponentVelocity(), 0.01f);
	absattr.forward = ConvertUnrealToFmodVector(UKismetMathLibrary::GetForwardVector(this->GetComponentRotation()));
	absattr.up = ConvertUnrealToFmodVector(UKismetMathLibrary::GetUpVector(this->GetComponentRotation()));

	attrs.absolute = absattr;

	dsp_pan->setParameterData(FMOD_DSP_PAN_3D_POSITION, &attrs, sizeof(FMOD_DSP_PARAMETER_3DATTRIBUTES_MULTI));

	group->set3DAttributes(&absattr.position, &absattr.velocity);
}

As stated, I cannot get the spatialization to work this way.
Basically, on BeginPlay I just create both DSPs on the master channel group now. The VoIP DSP simply copies the buffer coming from voice chat to the fmod channel’s buffer in the dspreadcallback event handler. This is working so far.
Then I also attach the Pan DSP to the channel group and set the Attenuation Settings once. After that, in each Tick event I update the 3D Position of the Pan DSP. The attenuation settings are exposed as variables to Blueprint and are set to FMOD’s default values for the DSP. The ConvertUnrealToFmodVector() function simply creates an FMOD_VECTOR and assigns the values of the FVector to it - and then optionally scaling it to convert to FMOD coordinates.
Nothing of all that currently has an effect and the Voice simply appears as a 2D Sound without any panning, attenuation etc.

Some questions I had on the way and couldn’t figure out:

Do I need to update the CoreSystem’s Listener’s position each frame? Or is that already covered in the Unreal Plugin?
Do I need to give the relative position to the listener or is the absolute position fine?
Do I need to define a listener myself at the beginning of runtime? In the documentation it says at least that it is attached to the camera by default.
Why does 3D Spatialization need the output mode of the DSP to be FMOD_DSP_PAN_MODE_SURROUND?

Connor_FMOD · June 19, 2024, 11:45pm

Thank you for all the info.

Could I grab your FMOD and Unreal versions?

stephen_tool1 · June 20, 2024, 8:09am

Sorry, sure:
UE5.3 and FMOD2.02.20

Connor_FMOD · June 24, 2024, 4:52am

Thank you for the code snippet. Unfortunately, I cannot get it working.

Would it be possible to get a copy of your project or a stripped-down version displaying the issue uploaded to your profile? Please note you must register a project with us before uploading files.

The plugin handles it as it is attached to the camera. Unreal Integration | User Guide - Listener. However, this can be manually overridden.

I would have to investigate this further sorry.

No, unless you want more than one listener in the map.

Where did you find this?

Thank you for the information. Have you had any luck implementing the other user’s solution?

stephen_tool1 · June 27, 2024, 9:49am

I have uploaded the project. To start, you should just need to double-click the uproject file. Once open you can PIE with 2 players and a listen server so that they are in the same session. When speaking you should hear your voice looping back (for the project I set the Unfocused Audio to full so you also hear the non-active client’s output).

The logic in the project happens in the OdinFmodAdapter source files. The relevant parts I already lined out above: BeginPlay, Update3DPosition and UpdateAttenSettings.

The project also includes the Odin part for easier reproduction. The copying of the voice chat buffer is handled in the OdinDSPReadCallback and the dspreadcallback functions.

Where did you find this?

Maybe I misinterpreted this: https://www.fmod.com/docs/2.01/api/effects-reference.html#2d-stereo-output-mode
So to clarify, FMOD_DSP_PAN_MODE_STEREO is fine?

Connor_FMOD · July 3, 2024, 6:20am

Hi,

Thank you for uploading that, it was really useful. I was able to get it running and hear mic input.

I am working on the issue but it has taken longer than I expected, I will hopefully have an answer for you by the end of the week.

Connor_FMOD · July 5, 2024, 5:06am

I appreciate your patience.

I hope I have found a solution.

I noticed we are missing the numoutputbuffer from the Odin DSP

FMOD_DSP_DESCRIPTION desc = { 0 };
desc.read = mReadCallback;
desc.numoutputbuffers = 1;
desc.userdata = this;

The issue might be adding the Odin DSP before the panner DSP meaning that it is not feeling the effects of the specialization. Could you please trying placing the DSP at the HEAD rather than the TAIL
```
if (group->addDSP(FMOD_CHANNELCONTROL_DSP_HEAD, dsp_objectpan) == FMOD_RESULT::FMOD_OK)
```

I found this spatialized audio for me.
There are also some asserts when ending PIE

LogFMOD: Error: C:\buildagent1\work\570ad728b1678096\core_api\src\fmod_systemi.cpp(853) - assertion: 'connectionsRemaining == 0' failed

Both DSPs are not being disconnected and released.

I hope this helps, please let me know if you encounter any other issues! Thank you for sharing this code, hopefully, it will help others in the future!

stephen_tool1 · July 8, 2024, 8:25am

Hi Connor,

thank you so much for looking into it! These two changes indeed make the spatialization plugin do something. Currently the direction does not seem to be calculated correctly - sound either goes fully to the left or right channel (but falloff seems correct for now). I will look into that, but maybe you already have an idea?

For example, what is the handedness and rotation of the coordinate system of FMOD in the Unreal Plugin? Is it the default for FMOD Studio, or is it set to the Unreal Coordinate System on the init call of the plugin?

EDIT: Okay, I could figure that out. But I don’t really get how to properly disconnect/remove/unregister/release a DSP.

This is what I do so far. I override the OnUnregister-Method of the Component - the code gets called, but it still prints the old error and also “Failed to release because unit is still attached. Use removeDSP function first.”

void UOdinFmodAdapter::OnUnregister()
{
	FMOD::Studio::System* System = IFMODStudioModule::Get().GetStudioSystem(EFMODSystemContext::Runtime);
	FMOD::System* CoreSystem = nullptr;
	System->getCoreSystem(&CoreSystem);

	CoreSystem->getMasterChannelGroup(&group);


	dsp_objectpan->disconnectAll(true, true);
	group->removeDSP(dsp_objectpan);
	mOdinDSP->disconnectAll(true, true);
	group->removeDSP(mOdinDSP);

	dsp_objectpan->release();
	mOdinDSP->release();

	Super::OnUnregister();
}

stephen_tool1 · July 9, 2024, 1:20pm

Okay, so it seems that in the OnUnregister() function of a Component, the Fmod Module is already destroyed. Is there a best practice to properly release the DSP? I do not seem to be able to find anything on that.

Connor_FMOD · July 15, 2024, 12:04am

Thank you for the code!

I am testing on my end too but running into some crashes. Could you please try adding these calls to the EndPlay() function, also checking if the System->isValid() in both functions. Hopefully one of them can release the DSP before the system is closed.

I will continue to test on my end. Just to confirm, you solved the specialization issues?

Again thank you for your patience.