DSP fader zeroing spacialised streaming sound

,

I’m creating a sound like this:

int NumChannels, SampleFrequency; // Passed in as function args.

FMOD::Studio::System* StudioSystem = IFMODStudioModule::Get().GetStudioSystem(EFMODSystemContext::Runtime);
check(StudioSystem != nullptr);
FMOD::System * CoreSystem = nullptr;
const FMOD_RESULT GetCoreSystemResult = System->getCoreSystem(&CoreSystem);
check(CoreSystem != nullptr);
check(GetCoreSystemResult == FMOD_OK);

FMOD::Sound* Result = nullptr;

FMOD_MODE Mode = FMOD_LOOP_NORMAL | FMOD_OPENUSER | FMOD_CREATESTREAM | FMOD_3D | FMOD_3D_WORLDRELATIVE;
FMOD_CREATESOUNDEXINFO SoundInfo;
FMemory::Memset(SoundInfo, 0);
SoundInfo.cbsize = sizeof(FMOD_CREATESOUNDEXINFO);
SoundInfo.numchannels = NumChannels;
SoundInfo.format = FMOD_SOUND_FORMAT_PCM16;
SoundInfo.decodebuffersize = 1024 * 3;	// This is from a sample, so I trust it to be reasonable.
SoundInfo.defaultfrequency = SampleFrequency;
SoundInfo.length = sizeof(int16) * SampleFrequency;
SoundInfo.pcmreadcallback = OnVoiceChatSoundReadDataCallback;
SoundInfo.userdata = static_cast<void*>(&UserChatInfo);
const FMOD_RESULT MakeSoundResult = CoreSystem->createSound(nullptr, Mode, &SoundInfo, &Result);

check(MakeSoundResult == FMOD_OK);

Then playing it like this:

FMOD::Channel* Channel = nullptr;
const FMOD_RESULT PlaySoundResult = CoreSystem->playSound(UserInfo.Sound, nullptr, false, &UserInfo.Channel);

check(PlaySoundResult == FMOD_OK);

FMOD_MODE Mode = 0;
const FMOD_RESULT GetVoiceChannelModeResult = Channel->getMode(&Mode);
check(GetVoiceChannelModeResult == FMOD_OK);
constexpr FMOD_MODE SettingsToNull = FMOD_2D | FMOD_3D | FMOD_3D_CUSTOMROLLOFF | FMOD_3D_INVERSEROLLOFF | FMOD_3D_INVERSETAPEREDROLLOFF | FMOD_3D_LINEARROLLOFF | FMOD_3D_LINEARSQUAREROLLOFF;
constexpr FMOD_MODE SettingsToSet = FMOD_3D | FMOD_3D_INVERSEROLLOFF;
Mode = Mode & ~SettingsToNull;
Mode = Mode | SettingsToSet;
const FMOD_RESULT SetVoiceChannelModeResult = ChannelToSetUp->setMode(Mode);
check(SetVoiceChannelModeResult == FMOD_OK);

float MinDistance = 0.0f;
float MaxDistance = 10000.0f; // Class settings - set in metres.
const FMOD_RESULT SetMinMaxDistanceResult = Channel->set3DMinMaxDistance(MinDistance, MaxDistance);
check(SetMinMaxDistanceResult == FMOD_OK);

uint8 VoipChannelPriority = 0;
const FMOD_RESULT SetPriorityResult = UserInfo.Channel->setPriority(VoipChannelPriority);
check(SetPriorityResult == FMOD_OK);

But I’m not able to hear the sound.

I’ve grabbed and logged every bit of information I can, checking that the modes and priorities are where I expect them to be, that the sound’s state is FMOD_OPENSTATE_PLAYING, that the channel isn’t virtual, paused, and has a volume set to 1, that the listener’s position and the sound source’s position is what I expect it to be. I’m totally confused. The audibility of the channel seems to be 0.0, when I read it .My only real clue is this:

int NumDsps = 0;
const FMOD_RESULT GetDSPNumResult = Channel->getNumDSPs(&NumDsps);
check(GetDSPNumResult == FMOD_OK);

for(int i=0;i<NumDsps;++i)
{
	FMOD::DSP* Dsp = nullptr;
	const FMOD_RESULT GetDSPResult = Channel->getDSP(i, &Dsp);
	check(GetDSPResult == FMOD_OK);
	check(Dsp != nullptr);

	FMOD_DSP_TYPE Type = FMOD_DSP_TYPE_UNKNOWN;
	const FMOD_RESULT GetDSPTypeResult = Dsp->getType(&Type);	
	check(GetDSPTypeResult == FMOD_OK);

	const FMOD_RESULT SetMeteringEnabledResult = Dsp->setMeteringEnabled(true,true);
	check(SetMeteringEnabledResult == FMOD_OK);

	if (Type == FMOD_DSP_TYPE_FADER)
	{
		float FaderValue = 0.0f;
		constexpr int BufferSize = 256;
		char GainValueStr[BufferSize];
		FMemory::Memset(GainValueStr, 0);
		const FMOD_RESULT GetGainResult = Dsp->getParameterFloat(FMOD_DSP_FADER_GAIN,&FaderValue,GainValueStr, BufferSize);
		check(GetGainResult == FMOD_OK);
		UE_LOG(LogTemp, Log, TEXT("DSP[%d]: Fader. Gain value=%3.2f"), i, FaderValue);

		FMemory::Memset(GainValueStr, 0);
		void* ResultPtr = nullptr;
		unsigned int ResultSize = 0;
		const FMOD_RESULT GetOverallGainResult = Dsp->getParameterData(FMOD_DSP_FADER_OVERALL_GAIN, &ResultPtr, &ResultSize, GainValueStr, BufferSize);
		check(GetOverallGainResult == FMOD_OK);
		FMOD_DSP_PARAMETER_OVERALLGAIN* DataPtr = static_cast<FMOD_DSP_PARAMETER_OVERALLGAIN*>(ResultPtr);
		UE_LOG(LogTemp, Log,TEXT("DSP[%d]: Overall gain: Linear=%3.2f, Additive=%3.2f"), i, DataPtr->linear_gain, DataPtr->linear_gain_additive);
	}

	FMOD_DSP_METERING_INFO InputInfo, OutputInfo;
	const FMOD_RESULT GetDSPMeteringResult = Dsp->getMeteringInfo(&InputInfo, &OutputInfo);
	check(GetDSPMeteringResult == FMOD_OK);

	const auto GetPeakLevel = [](const FMOD_DSP_METERING_INFO& Info)
	{
		float Max = 0.0f;
		for (short i = 0; i < Info.numchannels; ++i)
		{
			Max = FMath::Max(Max, Info.peaklevel[i]);
		}
		return Max;
	};

	UE_LOG(LogTemp, Log, TEXT(" DSP[%d]: DSP type=%d ; In=%3.2f Out=%3.2f"), i, Type, GetPeakLevel(InputInfo), GetPeakLevel(OutputInfo));
}

Using a generated white-noise sound I can see the data going in to the system and see that the OnVoiceChatSoundReadDataCallback is being called regularly to read the sound data. However, this bit of logging reveals:

  • 1 DSP - a fader
  • The gain on the fader is set to 0.0 (which is, I believe dB)
  • Overall linear gain is 1.0, with additive gain at 0.0f.
  • Peak level going into the fader is ~0.98 (but varies from 0.96 to 0.99).
  • Peak level coming out of the fader is always 0.0. (Why?)

I tried experimenting setting the gain on the fader to 1.0, just in case. In that case the overall gain is 1.12 linear, but the output is still 0.0.

I’m at a loss with where to go next with this investigation. Can anyone help?

Mini-update - If I set the mode to FMOD_2D it does produce sound although obviously I have no spatialisation in that case.

That is correct, this just means 0dB of gain is being applied by the fader.

This is probably because the Event/Channel doesn’t have its 3D attributes set.

I am not seeing any calls to EventInstance::set3DAttributes, can you please confirm you are calling that on your playing EventInstance, maybe somewhere in TickComponent?
If you use UFMODAudioComponent or AFMODAmbientSound we’ll keep the position updated for you, but to spatialize an event instance you need to keep the position updated manually. For example:

// If using an event instance to play your sound
FMOD_VECTOR attr = { { 0 } };
attr.position = FMODUtils::ConvertWorldVector(GetComponentTransform().GetLocation());
attr.up       = FMODUtils::ConvertUnitVector(GetComponentTransform().GetUnitAxis(EAxis::Z));
attr.forward  = FMODUtils::ConvertUnitVector(GetComponentTransform().GetUnitAxis(EAxis::X));
attr.velocity = FMODUtils::ConvertWorldVector(GetOwner()->GetVelocity());

StudioInstance->set3DAttributes(&attr);

// Or if using the core API only to play your sound
FMOD_VECTOR pos = FMODUtils::ConvertWorldVector(GetComponentTransform().GetLocation());
FMOD_VECTOR vel = FMODUtils::ConvertWorldVector(GetOwner()->GetVelocity());

UserInfo.Channel->set3DAttributes(&pos, &vel);

Thanks Jeff.

I do have an implementation which uses a UFMODAudioComponent. However, it’s not ideal as the component is attached to the player pawn, which means it gets destroyed and restarted as we server-travel, and and ideally we want the ability to use voice chat to persist through that.

Even then, we have an issue where the location of the UFMODAudioComponent never seems to update. I’ve found that in game the sound’s location is always the point the player spawned in, but it doesn’t update to the point they have moved to. So I’m trying to migrate to use the core APIs.

On the pawn’s tick we do this:

FVoiceChatUserInfo& UserInfo = GetVoiceChatUserInfo_UNSAFE(UserId);
UserInfo.Location = VoiceTransform.GetLocation();
UserInfo.Forward = VoiceTransform.GetRotation().GetForwardVector();
UserInfo.Up = VoiceTransform.GetRotation().GetUpVector();

UserInfo.bLocationDirty = true;

and then when sound data arrives we do this:

if (UserInfo.bLocationDirty)
{
	// Don't bother with the velocity. It's only used for doppler effects.
	static const FMOD_VECTOR Velocity{0.0f,0.0f,0.0f};
	const FVector UnrealLocation = UserInfo.Location;
	const FVector UnrealForward = UserInfo.Forward;

	const FMOD_VECTOR Location = FMODUtils::ConvertWorldVector(UnrealLocation);
	FMOD_VECTOR Forward = FMODUtils::ConvertUnitVector(UnrealForward);

	const FMOD_RESULT SetLocationResult = UserInfo.Channel->set3DAttributes(&Location, &Velocity);
	const FMOD_RESULT SetForwardResult = UserInfo.Channel->set3DConeOrientation(&Forward);

	if (SetLocationResult == FMOD_OK && SetForwardResult == FMOD_OK)
	{
		UserInfo.bLocationDirty = false;
	}
	else
	{
		// Report the error(s).
	}
}

Monitoring this, it is getting called. Additionally, debug mode I’m calling Channel->get3DAttributes() on the channel being used every frame and tracing the result as well as the location of any listeners on the core FMOD system.

Both traces give me the results I expect:

FMOD::System* System = /* Get the FMOD core system */;

auto GetFmodErrorString = [](FMOD_RESULT Result)
{
	return FString{ANSI_TO_TCHAR(FMOD_ErrorString(Result))};
};

auto VecToString = [](FMOD_RESULT Result, const FMOD_VECTOR& Vec)
{
	if (Result != FMOD_OK)
	{
		return FString::Printf(TEXT("[CouldNotGetVector=%s]"), *GetFmodErrorString(Result));
	}
	return FString::Printf(TEXT("[%3.2f,%3.2f,%3.2f]"), Vec.x, Vec.y, Vec.z);
};

auto IsModeToString = [](FMOD_RESULT GetModeResult, FMOD_MODE Mode, FMOD_MODE Reference) -> FString
{
	if (Mode == 0)
	{
		return FString::Printf(TEXT("[CouldNotGetMode=%s]"), *GetFmodErrorString(GetModeResult));
	}
	const auto Masked = Mode & Reference;
	const bool bIsTrue = Masked != 0;
	return bIsTrue ? TEXT("true") : TEXT("false");
};

FMOD_MODE Mode = 0;
const FMOD_RESULT GetModeResult = UserInfo.Channel->getMode(&Mode);

FMOD_VECTOR SoundLocation, SoundVelocity;
const FMOD_RESULT Get3DAttr = UserInfo.Channel->get3DAttributes(&SoundLocation, &SoundVelocity);

FString Msg = FString::Printf(TEXT("VOIP for %s got ChannelAttributes=[Loc=%s Vel=%s HeadSpace=%s WorldSpace=%s]"),
	*UserId,
	*VecToString(Get3DAttr, SoundLocation),
	*VecToString(Get3DAttr, SoundVelocity),
	*IsModeToString(GetModeResult, Mode, FMOD_3D_HEADRELATIVE),
	*IsModeToString(GetModeResult, Mode, FMOD_3D_WORLDRELATIVE));

int NumListeners = 0;
const FMOD_RESULT GetNumListenersResult = System->get3DNumListeners(&NumListeners);
check(GetNumListenersResult == FMOD_OK);
Msg = FString::Printf(TEXT("%s\n    Found %d %s%s"),
	*Msg,
	NumListeners,
	NumListeners == 1 ? TEXT("listener") : TEXT("listeners"),
	NumListeners == 0 ? TEXT("") : TEXT(":"));
for (int i = 0; i < NumListeners; ++i)
{
	FMOD_VECTOR Location, Velocity, Forward, Up;
	const FMOD_RESULT GetListenerAttrResult = System->get3DListenerAttributes(i, &Location, &Velocity, &Forward, &Up);
	check(GetListenerAttrResult);
	Msg = FString::Printf(TEXT("%s\n        %d: [Loc=%s Fwd=%s]"),
		*Msg,
		i,
		*VecToString(GetListenerAttrResult, Location),
		*VecToString(GetListenerAttrResult, Forward));
}

The following trace in the logs reports locations (and listener forward vectors) that vary per-frame as expected:

VOIP for BP_FmodTestActor_2 got ChannelAttributes=[Loc=[-18.21,0.87,-18.75] Vel=[0.00,0.00,0.00] HeadSpace=false WorldSpace=true]
    Found 1 listener:
	0: [Loc=[-18.62,1.88,-19.44] Fwd=[0.20,-0.71,0.67]]

To confirm: I am calling set3DAttributes regularly (every frame while there is a pawn) and the tracing indicates that the listener and channel locations are correct.

Thank you for confirming, it certainly looks like the Channel is being given a position correctly.

I have a few more ideas on things to debug:

  • Pass in the FMOD_3D_INVERSEROLLOFF mode to your System::createSound call. This will rule out any issues with world scaling because sounds will always be audible in this mode, even if the listener is outside the Sound’s maximum range. If you can hear the Sound with this mode, the issue is probably with scaling.
  • Try calling Channel::getAudibility. I expect this to be 0 based on what you’ve described, but it would be good to confirm.

Hi Jeff,

Thanks for your advice on this one. I had been checking audibility and it was 0, but I hadn’t any idea why.

In any case: I switched the implementation to use an EventDescription and an EventInstance instead (partially because our audio team were providing appropriate assets to use which they could set up in the Studio that way) and it’s working much better … sort of.

I can call get3Dattributes() every frame and log the result, and also do similar with the listener. So I can see that the event’s 3D attributes are updating position, forward, and up vectors as I expect, and I can also see that the listener’s location is updating as expecting.

And the spatialisation panning works, but the volume does not. The volume is loudest at whatever point we first play the sound, but moving the location does not move that point. It’s like the direction is calculated correctly, but the distance is calculated using only the initial location instead of the current one.

Our audio engineers insist the event is correctly set up, but I don’t really have any way to actually check the setup from where I am.

(EDIT: I’ve gone and inspected all the parameters on the event in real time using the code APIs to get them. Looks like there is a game controlled one that’s doing the fade. Looks like this one is on our end.)

1 Like

Thanks for the update, I can see how an automated fader could be causing unexpected spatialization. Please let me know if you run into any other issues.