Creating a custom plugin

[Leah_FMOD]

@apfelbaum

Looking at the examples there are a few plugins like “generate_tone” which completely setup channel, plugins or sounds at startup of the application up to the main loop where everything is updated. Is there some kind of way to do this in unreal?

Yes, but it depends on what exactly you want to do, and whether you’re just using the Core API like the generate_tone example is, or the Studio API.

If you were to just use the Core API, you can essentially do the same thing that generate_tone does: create your DSP/Sound objects, and then play them on a Channel, adjusting the DSP/Channel parameters as needed. You would have to manage the lifetimes of these objects yourself.

If you have a Studio event set up with your granular synth DSP and looped samples like you have, it’s a little more complex. The main point of issue is taking the RPM output by your granular synth and inputting to your looped samples, which you will need to directly access the DSP in order to do. Setting an Event Callback is probably the best way to do this. You should be able to use FMOD_STUDIO_EVENT_CALLBACK_PLUGIN_CREATED to get access to your granular synth DSP, pass a handle to it out of the callback as User Data, and set/get the RPM as needed.

[apfelbaum]

As I use only the FMOD plugin for Unreal it is the Studio API then.

That sounds great. Can I somehow send you the VS project of the plugin? Whatever I do, I cannot get the DSP plugin correctly in code, it is always recognized as sound in a channel and I cannot cast it around somehow. As soon that works I will definitely try that out.

[Leah_FMOD]

@apfelbaum

Can I somehow send you the VS project of the plugin? Whatever I do, I cannot get the DSP plugin correctly in code, it is always recognized as sound in a channel and I cannot cast it around somehow. As soon that works I will definitely try that out.

Do you receive any warnings or errors from the FMOD API function you’re calling, and from FMOD in general? I’d recommend changing FMOD’s “Logging Level” settings in Unreal to LEVEL_LOG and seeing what errors occur, if you haven’t already.

If that doesn’t reveal any extra details, you can upload your plugin’s VS project and the code you’re using to play it in Unreal to the Uploads tab of your FMOD User Profile.

[apfelbaum]

No I don’t receive any warnings in this regard. I uploaded the project, I hope .zip files are not a problem for you
It contains the VS Project, the compiled plugin with .js styling and an example file. I also print out a little to the log. I think of interest for you are only the main files as they contain the FMOD stuff. Thank you for checking it out.

Btw., I noticed the AudioMotors 2 plugin has the same issue as my plugin.

Edit.: My problem is finding the plugin. This kind of plugin is on a Channel, not a DSPConnection or a ChannelGroup right? I tried Dsp Inputs, Outputs, i searched all channels, sounds and connections but it is nowhere to be found. It’s a miracle to me.

Oh almost forgot. Here the code to check find the DSP. The event is just the plugin on a single track on a 3d timeline.

header file

// Fill out your copyright notice in the Description page of Project Settings.

#pragma once

#include "CoreMinimal.h"
#include "Components/SceneComponent.h"
#include "FMODAudioComponent.h"
#include "EngineTest.generated.h"

UCLASS( ClassGroup=(Custom), meta=(BlueprintSpawnableComponent) )
class GINSUTEST_API UEngineTest : public USceneComponent
{
	GENERATED_BODY()

public:	
	UEngineTest();

public:	
	virtual void TickComponent(float DeltaTime, ELevelTick TickType, FActorComponentTickFunction* ThisTickFunction) override;

	UPROPERTY(EditAnywhere)
		class UFMODEvent* Event;

	UPROPERTY()
		class UFMODAudioComponent* AudioComp;
protected:
	void ExtractGroup(FMOD::ChannelGroup* Group);
	void ExtractChannel(FMOD::Channel* Channel);
	void ExtractDSPs(FMOD::ChannelControl* ChannelControl);
	void ExtractSounds(FMOD::Channel* Channel);
	void GetDSPProperties(FMOD::DSP* DSP);
};

source file

// Fill out your copyright notice in the Description page of Project Settings.


#include "EngineTest.h"


UEngineTest::UEngineTest()
{
	PrimaryComponentTick.bCanEverTick = true;
}

void UEngineTest::TickComponent(float DeltaTime, ELevelTick TickType, FActorComponentTickFunction* ThisTickFunction)
{
	Super::TickComponent(DeltaTime, TickType, ThisTickFunction);

	if (AudioComp == nullptr) {
		FAttachmentTransformRules AttachmentRules
		(
			EAttachmentRule::SnapToTarget,
			EAttachmentRule::SnapToTarget,
			EAttachmentRule::KeepRelative,
			true
		);
		AudioComp = NewObject<UFMODAudioComponent>(this, UFMODAudioComponent::StaticClass());
		AudioComp->RegisterComponent();
		AudioComp->AttachToComponent(GetOwner()->GetRootComponent(), AttachmentRules);
		AudioComp->SetEvent(Event);
		AudioComp->Play();
		AudioComp->SetParameter(TEXT("Rpm"), 1500.0f);
	}

	const FMOD::Studio::EventInstance* Instance = AudioComp->StudioInstance;

	if (Instance == nullptr) return;

	FMOD_STUDIO_PLAYBACK_STATE PlaybackState;
	FMOD_RESULT Result = Instance->getPlaybackState(&PlaybackState);

	if (Result != FMOD_OK || PlaybackState != FMOD_STUDIO_PLAYBACK_PLAYING) return;

	FMOD::ChannelGroup* ChannelGroup;
	Result = Instance->getChannelGroup(&ChannelGroup);
	if (Result != FMOD_OK) return;

	ExtractGroup(ChannelGroup);
}

void UEngineTest::ExtractGroup(FMOD::ChannelGroup* Group)
{
	FMOD_RESULT Result;
	//ExtractDSPs(Group);

	int32 NumChannels = 0;
	Result = Group->getNumChannels(&NumChannels);
	if (Result == FMOD_OK)
	{
		for (int32 i = 0; i < NumChannels; ++i)
		{
			FMOD::Channel* Channel;
			Result = Group->getChannel(i, &Channel);
			if (Result != FMOD_OK) continue;

			ExtractChannel(Channel);
		}
	}

	int32 NumGroups = 0;
	Result = Group->getNumGroups(&NumGroups);
	if (Result == FMOD_OK)
	{
		for (int32 i = 0; i < NumGroups; ++i)
		{
			FMOD::ChannelGroup* SubGroup;
			Result = Group->getGroup(i, &SubGroup);
			if (Result != FMOD_OK) continue;

			ExtractGroup(SubGroup);
		}
	}
}

void UEngineTest::ExtractChannel(FMOD::Channel* Channel)
{
	ExtractDSPs(Channel);
	ExtractSounds(Channel);
}

void UEngineTest::ExtractDSPs(FMOD::ChannelControl* ChannelControl)
{
	GEngine->AddOnScreenDebugMessage(-1, 0.0f, FColor::White, FString::Printf(TEXT("DSP")));
	int32 NumDSPs = 0;
	FMOD_RESULT Result = ChannelControl->getNumDSPs(&NumDSPs);
	if (Result == FMOD_OK)
	{
		for (int32 i = 0; i < NumDSPs; ++i)
		{
			FMOD::DSP* DSP;
			Result = ChannelControl->getDSP(i, &DSP);
			if (Result != FMOD_OK) continue;
			GetDSPProperties(DSP);
		}
	}
}

void UEngineTest::ExtractSounds(FMOD::Channel* Channel)
{
	FMOD::Sound* Sound;
	FMOD_RESULT Result = Channel->getCurrentSound(&Sound);
	if (Result != FMOD_OK) return;

	GEngine->AddOnScreenDebugMessage(-1, 0.0f, FColor::White, FString::Printf(TEXT("Found a sound")));
}

void UEngineTest::GetDSPProperties(FMOD::DSP* DSP)
{
	char name[50];
	uint32* Version = 0;
	int32 Channels = 0, ConfigWidth = 0, ConfigHeight = 0;
	FMOD_RESULT Result = DSP->getInfo(name, Version, &Channels, &ConfigWidth, &ConfigHeight);
	if (Result != FMOD_OK) return;

	const FString DSPName(name);
	GEngine->AddOnScreenDebugMessage(-1, 0.0f, FColor::White, FString::Printf(TEXT("%s"), *DSPName));

	int32 NumInputs;
	Result = DSP->getNumInputs(&NumInputs);
	if (Result == FMOD_OK && NumInputs > 0)
	{
		for (int32 i = 0; i < NumInputs; ++i)
		{
			FMOD::DSP* InDSP;
			FMOD::DSPConnection* DSPConnection;
			Result = DSP->getInput(i, &InDSP, &DSPConnection);
			if (Result == FMOD_OK)
			{
				Result = DSPConnection->getInput(&InDSP);
				if (Result != FMOD_OK) continue;
				GEngine->AddOnScreenDebugMessage(-1, 0.0f, FColor::White, FString::Printf(TEXT("DSPConnection")));
				GetDSPProperties(InDSP);
			}
		}
	}
}

There is one sound found, i dont know why. But calling any method on it fails. Get name fails with FMOD_ERR_INVALID_HANDLE.

[Leah_FMOD]

Thanks for uploading your code!

I can see what you mean by having issues with the plugin when traversing the DSP graph. My expectation that you could call setParameter functions on the “FMOD Resampler” was incorrect, so you have my apology for that - it seems that the actual custom DSP is stored as an independent subgraph that is connected to the resampler, and cannot be accessed by the normal means of traversing the DSP graph.

That said, using an event callback will give you direct access to your Plugin DSP fairly simply. I have created a struct as follows in the UE header to pass to the event as user data:

    struct UserData {
        FMOD::DSP* pluginDSP;
    };

    const UserData userData;

As well as the following static event callback, which retrieves the DSP when it is created, and passes it to the user data struct:

FMOD_RESULT F_CALLBACK UEngineTest::EventCallback(FMOD_STUDIO_EVENT_CALLBACK_TYPE type, FMOD_STUDIO_EVENTINSTANCE* event, void* parameters) {
	if (type == FMOD_STUDIO_EVENT_CALLBACK_PLUGIN_CREATED) {
		UserData* userData;
		((FMOD::Studio::EventInstance*)event)->getUserData((void**) &userData);
		userData->pluginDSP = (FMOD::DSP*)(((FMOD_STUDIO_PLUGIN_INSTANCE_PROPERTIES*)parameters)->dsp);
	}
	return FMOD_OK;
}

I have appended the following code to the end of your code in the if (AudioComp == nullptr) check in UEngineTest::TickComponent(), which assigns the User Data struct and Callback to the AudioComponent’s underlying Event Instance:

if (AudioComp == nullptr) {
    //....
    AudioComp->StudioInstance->setCallback(EventCallback);
    void* userDataPointer = (void*)&userData;
    AudioComp->StudioInstance->setUserData(userDataPointer);
}

And finally, I’ve set the RPM parameter on the retrieved plugin DSP at the end of UEngineTest::TickComponent() with the following code:

//...
if (userData.pluginDSP != nullptr) {
    userData.pluginDSP->setParameterFloat(0, 5000);
}

With this code, I am successfully able to retrieve the plugin DSP and set all parameters on it without any issue. Please give a shot on your and see whether you can access your DSP plugin in code - if so, you can omit traversing the DSP graph entirely.

[apfelbaum]

This worked like a charm. I must distinguish the plugins now by a boolean I guess as there are 2 per engine, one for acceleration and one for deceleration.

Is it possible to have a callback on the dsp which is called after the dsp “process” callback, not Unreal Engines tick? Because I have to set the looped samples rpm value directly after the dsp is finished processing.

Best regards.

Edit.: I noticed something strange: The Unreal Engine Event is lower pitched than in FMOD. I don’t change the sound at all afterwards in this test project, it is purely the DSP that outputs the audio file at 48khz.

IGNORE THE PREVIOUS EDIT, THAT WAS MY FAULT.

Edit.: With your help I am now able to have a very smooth blending between the granular synth and the looped samples. In this examples you can hear the 2 granular synths for on- and offload and also 6 looped samples for on- and offload.

engine sound

One problem is here which is due to the way i update the looped samples rpm. On Unreal Engines Tick I get the dsp’s output rpm and set the looped samples rpm. so the looped samples rpm is always set 1 mixer tick later than the dsp outputs its samples. so whenever you change the rpm too fast you hear a slight phasing as the looped samples update 1 buffer later and it is again out of sync. That is the reason I ask wether I can register eg. a callback in the dsp that is called at the end of the process function so i can update the event float parameter.

[Leah_FMOD]

@apfelbaum

On Unreal Engines Tick I get the dsp’s output rpm and set the looped samples rpm. so the looped samples rpm is always set 1 mixer tick later than the dsp outputs its samples. so whenever you change the rpm too fast you hear a slight phasing as the looped samples update 1 buffer later and it is again out of sync. That is the reason I ask wether I can register eg. a callback in the dsp that is called at the end of the process function so i can update the event float parameter.

The easiest way to handle this would probably be to pass a function pointer to your DSP as user data, which you then call from within the DSP at the end of the process call to set the looped samples RPM. This should have the effect of “cutting out the middleman” so to speak, in that you shouldn’t have to wait for an additional mixer tick.

Here’s a rough code snippet demonstrating creating a struct and assigning a function pointer to it:

struct Data
{
   void (*postmix)();
};

Data data = {0};
data.postProcess = MyCallback;
dsp->setUserData(data);

And the calling the function pointer from within your DSP’s process callback:

FMOD_RESULT process(FMOD_DSP_STATE *state, ...)
{
    //...
    //existing process callback code here
    //...
   getUserData()->postProcess();
   return FMOD_OK;
}

[apfelbaum]

So I had to laugh a bit because at first It didn’t work for me like that. I will share my solution here but maybe you have a better one. It is so ridiculous to me but it works and it sounds perfect now:

This is in the DSP

(this struct is mirrored in unreal)

This here is where the DSP callback created is in Unreal:

FMOD_RESULT F_CALLBACK EventCallback(FMOD_STUDIO_EVENT_CALLBACK_TYPE type, FMOD_STUDIO_EVENTINSTANCE* event, void* parameters) {
	if (type == FMOD_STUDIO_EVENT_CALLBACK_PLUGIN_CREATED) {
		FFMODUserData* userData;
		FMOD::Studio::EventInstance* EventInstance = reinterpret_cast<FMOD::Studio::EventInstance*>(event);
		FMOD_RESULT Result = EventInstance->getUserData(reinterpret_cast<void**>(&userData));
		const FMOD_STUDIO_PLUGIN_INSTANCE_PROPERTIES* Properties = static_cast<FMOD_STUDIO_PLUGIN_INSTANCE_PROPERTIES*>(parameters);
		FMOD::DSP* DSP = reinterpret_cast<FMOD::DSP*>(Properties->dsp);

		if (OutputParams->bIsDeceleration)
		{
			userData->GinsuDecel = DSP;
			OutputParams->PostProcess = PostProcessDecel;
			OutputParams->EventInstance = EventInstance;
			OutputParams->parameterName = "AEMS_DECEL_RPM";
			userData->DecelParams = OutputParams;
		}
		else
		{
			userData->GinsuAccel = DSP;
			OutputParams->PostProcess = PostProcessAccel;
			userData->AccelParams = OutputParams;
			OutputParams->EventInstance = EventInstance;
			OutputParams->parameterName = "AEMS_ACCEL_RPM";
		}
	}
	return FMOD_OK;
}

…and this is also in Unreal. Basicly I have to pass around the EventInstance, super duper spaghetti.

You can here how clean the samples now overlap, pretty much no phasing:

engine

Any ideas how to do it better? If not I would mark this as resolved. Thank you for this awesome journey.

[Leah_FMOD]

It sounds great! Due to the somewhat complex requirement of passing the RPM around, I suspect that some spaghetti-ness unavoidable. Just using a function pointer may be a little cleaner than passing the whole Event Instance, but other than that, I don’t have any further suggestions.

[Altarf82]

Hi Apfelbaum,
I hope you are ok. I am also big fan of older driving games, in my case Richard Burns Rally.
I really need your help with FMOD please.
I am currently creating rally game mod of RBR (main focus realistic physics see here https://www.youtube.com/watch?v=3HhC_Heb1Gc&t=40s) and I would love to implement the FMOD to it as it would be awesome. Please email me at przemek-guminski@wp,pl Thank you in advance

[Leah_FMOD]

@Altarf82

I hope you are ok. I am also big fan of older driving games, in my case Richard Burns Rally.
I really need your help with FMOD please.
I am currently creating rally game mod of RBR (main focus realistic physics see here https://www.youtube.com/watch?v=3HhC_Heb1Gc&t=40s) and I would love to implement the FMOD to it as it would be awesome.

Generally speaking, specific support for modding falls outside the scope of support beyond assisting with general API questions/issues - if there’s a modding community for RBR, I would recommend getting in touch with them for assistance. If you do have any specific questions, feel free to create a thread for them or reply to existing threads if relevant, but we heavily discourage spamming similar messages in multiple unrelated threads, so please refrain from doing so.