Creating a custom plugin

[apfelbaum]

Hi,
I just started creating my own plugin. I guess I have the includes and dll. linking for the fmod studio apu correctly setup. What I want to do now is create a granular synth on my own.
The general idea is to have a table which describes the sample in an audio file (num samples, sample length etc.). All the logic to calculate the correct samples is already done given an input parameter (spoiler alert, it is a GS for car engine sounds so the input parameter is the RPM value).
For this I need to directly set the samples that FMOD should play with additional pitching. Is that possible in FMOD and where should I start? I have a bit trouble reading just your docs and looking at the example plugins.

Edit.: Idea is to put the plugin like the AudioMotors plugin into the timeline and adjust the RPM parameter with curves. The audio file can be drag and dropped onto the plugin. It can be whatever FMOD supports. The arrays of data could pasted manually or saved in the header of the audio file.

Best regards

[Leah_FMOD]

A custom DSP plugin will allow you to accomplish what you want - If your DSP has zero input channels, it is treated as an instrument, and can be placed on a sheet in an event. If you specify that your DSP has a data parameter of type DSP_PARAMETER_DATA_TYPE_USER, the default Studio UI component for a data parameter will allow you to browse for or drag and drop an audio file into the DSP effect. I’d recommend taking a look at the Plugin Reference documentation for more info, and the API examples in your FMOD Studio API install, specifically the “fmod_gain” and “fmod_noise” examples.

I’m a little unclear as to what you’re trying to do with regards to the table of sample info and additional pitching that you’ve described - would you mind elaborating on exactly what you mean?

[apfelbaum]

Hi,
thank you for your answer. So the idea is to have a 2d array with the audio file in it where you access the first layer with the rpm and the second layer contains the audio samples at this position (the grain). With some calculations these samples are pulled out and returned to be played by FMOD. As the array of grains won’t contain all possible rpm values the pulled out grains are pitched slightly to match the rpm. Say you have an input value of 5100 rpm but only grains at 5050 and 5200. then you pull out the grains at 5050 and pitch them a bit up. This calculation is also already done.

[Leah_FMOD]

That makes sense, thanks for explaining in more detail.

As an aside, I’ve done a little more research into custom DSP UI components, specifically the ability to make use of default instrument UI components (i.e. playlist) when creating a custom DSP instrument, which would likely make adding samples to your DSP more convenient. However, after discussion with one of our programmers, we’ve come to the conclusion that this functionality isn’t exposed, so I’ve added it to our internal feature/improvement tracker.

That said, the default UI component for the user data parameter, DataDrop, should be fine for your needs. If you’re not using too many individual samples, then you could just use an arbitrary amount of data parameters to assign samples - though obviously with too many samples this would cause some UI clutter. If you’re planning to have a single larger audio file with table data saved into the header, as you’ve stated that you could do, a single data parameter that you use for the whole file would be fine, as you could seek/chop the whole file as needed in your custom plugin .dll.

[apfelbaum]

So I think I got the DataDrop working. But how do I get the data so I can read the binary? Inside the header of the file that I will drop I have all necessary data to define the sample rate, num samples etc. and afterwards I have a huge array of samples.

FMOD_RESULT F_CALLBACK Ginsu_Synth_DspSetParamData(FMOD_DSP_STATE* dsp, int index, void* data, unsigned int length)
{
	GinsuSynth* state = (GinsuSynth*)dsp->plugindata;
	FMOD_DSP_LOG(dsp, FMOD_DEBUG_LEVEL_ERROR, "Ginsu_Synth_DspSetParamData", "Text", "");



	return FMOD_ERR_INVALID_PARAM;
}

[Leah_FMOD]

When using FMOD_DSP_SETPARAM_DATA_CALLBACK, the void pointer data argument is a pointer to the binary data that is passed to the DSP (in this case via DataDrop). The length of the binary data is specified by the length argument. You’ll want to cast data to the appropriate type so you can access and operate on it as desired.

[apfelbaum]

Okay I have a little problem with the .plugin.js. The Datadrop is not visible and the frequency randomization.

this is the result:

this is the script:

studio.plugins.registerPluginDescription("Ginsu Synth", {
    companyName: "Custom",
    productName: "Ginsu Synthesizer",

    deckUi: {
        deckWidgetType: studio.ui.deckWidgetType.Layout,
        layout: studio.ui.layoutType.HBoxLayout,
        spacing: 18,
        items: [
            {
                deckWidgetType: studio.ui.deckWidgetType.Layout,
                layout: studio.ui.layoutType.VBoxLayout,
                spacing: 12,
                items: [
                    { deckWidgetType: studio.ui.deckWidgetType.Pixmap, filePath: __dirname + "/ginsu_synth_logo.png" },
                    { deckWidgetType: studio.ui.deckWidgetType.DataDrop, binding: "Ginsu Synth Data", fileNameFilters: "*.gin" },
                ],
            },
            {
                deckWidgetType: studio.ui.deckWidgetType.Layout,
                layout: studio.ui.layoutType.GridLayout,
                spacing: 12,
                items: [
                    { deckWidgetType: studio.ui.deckWidgetType.Dial, color: "#DF171A", row: 0, column: 0, binding: "Rpm", },
                    { deckWidgetType: studio.ui.deckWidgetType.Dial, color: "#FFFFFF", row: 1, column: 0, binding: "Idle Rpm", },
                    { deckWidgetType: studio.ui.deckWidgetType.Dial, color: "#DF171A", row: 1, column: 1, binding: "Redline Rpm", },
                    { deckWidgetType: studio.ui.deckWidgetType.Dial, color: "#DF171A", row: 1, column: 2, binding: "Frequency Randomization", },
                ],
            },
        ],
    },
});

and this is in code

extern "C"
{
	F_EXPORT FMOD_DSP_DESCRIPTION* F_CALL FMODGetDSPDescription()
	{
		FMOD_DSP_INIT_PARAMDESC_FLOAT(p_rpm, "Rpm", "rpm", "0 to 10.000. Default = 1000", GINSU_SYNTH_PARAM_RPM_MIN, GINSU_SYNTH_PARAM_RPM_MAX, GINSU_SYNTH_PARAM_RPM_DEFAULT);
		FMOD_DSP_INIT_PARAMDESC_FLOAT(p_idle_rpm, "Idle Rpm", "rpm", "0 to 5.000. Default = 1000", GINSU_SYNTH_PARAM_IDLE_RPM_MIN, GINSU_SYNTH_PARAM_IDLE_RPM_MAX, GINSU_SYNTH_PARAM_IDLE_RPM_DEFAULT);
		FMOD_DSP_INIT_PARAMDESC_FLOAT(p_redline_rpm, "Redline Rpm", "rpm", "5.000 to 10.000. Default = 10000", GINSU_SYNTH_PARAM_REDLINE_RPM_MIN, GINSU_SYNTH_PARAM_REDLINE_RPM_MAX, GINSU_SYNTH_PARAM_REDLINE_RPM_DEFAULT);
		FMOD_DSP_INIT_PARAMDESC_FLOAT(p_frequency_randomization, "Frequency Randomization", "rpm", "0 to 100. Default = 0", GINSU_SYNTH_PARAM_FREQUENCY_RANDOMIZATION_MIN, GINSU_SYNTH_PARAM_FREQUENCY_RANDOMIZATION_MAX, GINSU_SYNTH_PARAM_FREQUENCY_RANDOMIZATION_DEFAULT);
		FMOD_DSP_INIT_PARAMDESC_DATA(p_ginsu_synth_data, "Ginsu Synth Data", ".gin file", "Some description", GINSU_SYNTH_PARAM_DATATYPE_DEFAULT);
		return &Ginsu_Synth_Desc;
	}
}

when i don’t use the .js all controls are there but pretty ugly aligned.

Best regards.

Edit.: Found out, the name of the control shouldn’t be too long.

[apfelbaum]

Okay so I have it now. The plugin is almost finished I think.

What I have here is the number of samples (which size is actually variable, i just gave it 512 slots for testing). The number of samples depends on the rpm of the car and is calculated. The samples are in int16_t and have a samplerate of 32khz. How do I get it into FMOD correctly?

Edit.:

Found out when I restructure the function like this I have output.

Do I have to change anything else? Is it correct that the buffer float are normalized to 1.0?

[apfelbaum]

synth

You can hear the first car sound being pitched too high because its sample rate is lower than fmods sample rate, the second sample rate is matching and sounds correct.

I fixed all bugs regarding the buffer. Now I just need to know how to set the samplerate for the plugin if that is possible. Or do I have to resample myself?

[Leah_FMOD]

Happy to see you’ve managed to figure most of it out!

Unfortunately, there’s no way to get the FMOD system to resample the DSP output for you, so you will have to resample the output yourself.

[apfelbaum]

I guess you don’t have a function for resampling in built that you can expose so one can call it in a DSP?

[Leah_FMOD]

@apfelbaum

I guess you don’t have a function for resampling in built that you can expose so one can call it in a DSP?

Unfortunately, there isn’t one, though I’ve noted your interest on our internal feature/improvement tracker.

There is Channel::setFrequency which (essentially) adds a resampler to a channel, but that function isn’t available to DSPs. When using the API, you could use the function on a channel that the DSP is on, but that would involve passing in either the channel or entire system to the DSP as user data - something that isn’t possible in Studio, as you have no way to access a handle to the system or channel.

[apfelbaum]

In the end I want this to be in Unreal Engine 5.2.

Btw., your help is super good and I love the support.

Edit.: I see by recursively looking for the channelgroups in Unreal Engine I am able to find all DSPs in channels. But somehow only “Channel Fader/ChanGroupFader/FMOD Pan” is found, not the Plugin instrument(s). Is there another way to find these?

void UGinsuSoundTest::TickComponent(float DeltaTime, enum ELevelTick TickType, FActorComponentTickFunction* ThisTickFunction)
{
	Super::TickComponent(DeltaTime, TickType, ThisTickFunction);

	if (AudioComp == nullptr) return;
	FMOD::Studio::EventInstance* Instance = AudioComp->StudioInstance;
	if (Instance == nullptr) return;

	FMOD_STUDIO_PLAYBACK_STATE PlaybackState;
	FMOD_RESULT Result = Instance->getPlaybackState(&PlaybackState);
	if (Result != FMOD_OK || PlaybackState != FMOD_STUDIO_PLAYBACK_PLAYING) return;

	FMOD::ChannelGroup* ChannelGroup;
	Result = Instance->getChannelGroup(&ChannelGroup);
	if (Result != FMOD_OK) return;

	ColorIndex = 0;
	ExtractGroup(ChannelGroup);
}

void UGinsuSoundTest::ExtractGroup(FMOD::ChannelGroup* Group)
{
	MyColor = FColor::MakeRandomSeededColor(ColorIndex++);
	int32 NumDSPs = 0;
	FMOD_RESULT Result = Group->getNumDSPs(&NumDSPs);

	if (Result == FMOD_OK)
	{
		for (int32 j = 0; j < NumDSPs; ++j)
		{
			FMOD::DSP* DSP;
			Result = Group->getDSP(j, &DSP);
			if (Result != FMOD_OK) continue;

			ExtractDSP(DSP);
		}
	}

	int32 numChannels = 0;
	Result = Group->getNumChannels(&numChannels);
	if (Result == FMOD_OK)
	{
		for (int32 i = 0; i < numChannels; ++i)
		{
			FMOD::Channel* Channel;
			Result = Group->getChannel(i, &Channel);
			if (Result != FMOD_OK) continue;

			ExtractChannel(Channel);
		}
	}

	int32 NumGroups = 0;
	Result = Group->getNumGroups(&NumGroups);
	if (Result == FMOD_OK)
	{
		for (int32 i = 0; i < NumGroups; ++i)
		{
			FMOD::ChannelGroup* ChildGroup;
			Result = Group->getGroup(i, &ChildGroup);
			if (Result != FMOD_OK) continue;

			ExtractGroup(ChildGroup);
		}
	}
}

void UGinsuSoundTest::ExtractChannel(FMOD::Channel* Channel) const
{
	int32 numDSPs = 0;
	FMOD_RESULT Result = Channel->getNumDSPs(&numDSPs);
	if (Result == FMOD_OK)
	{
		for (int32 j = 0; j < numDSPs; ++j)
		{
			FMOD::DSP* DSP;
			Result = Channel->getDSP(j, &DSP);
			if (Result != FMOD_OK) continue;

			ExtractDSP(DSP);
		}
	}
	FMOD::Sound* Sound;
	Result = Channel->getCurrentSound(&Sound);
	if (Result == FMOD_OK)
	{
		ExtractSound(Sound);
	}

	float Frequency = -1.0f;
	Result = Channel->getFrequency(&Frequency);
	GEngine->AddOnScreenDebugMessage(-1, 0.0f, MyColor, FString::Printf(TEXT("Frequency= %fHz"), Frequency));
}

void UGinsuSoundTest::ExtractDSP(FMOD::DSP* DSP) const
{
	char name[50];
	uint32* Version = 0;
	int32 Channels = 0, ConfigWidth = 0, ConfigHeight = 0;
	FMOD_RESULT Result = DSP->getInfo(name, Version, &Channels, &ConfigWidth, &ConfigHeight);
	if (Result == FMOD_OK)
	{
		FString DSPName(name);
		GEngine->AddOnScreenDebugMessage(-1, 0.0f, MyColor, FString::Printf(TEXT("DSPName= %s, Channels= %i"), *DSPName, Channels));
	}
	else
	{
		GEngine->AddOnScreenDebugMessage(-1, 0.0f, FColor::Red, FString::Printf(TEXT("DSP found but without info")));
	}
}

void UGinsuSoundTest::ExtractSound(FMOD::Sound* Sound) const
{
	constexpr int32 SoundNameLength = 32;
	char SoundName[SoundNameLength];
	FMOD_RESULT Result = Sound->getName(SoundName, SoundNameLength);
	if (Result == FMOD_OK)
	{
		FString str = GetNullTerminatedString(SoundName, SoundNameLength);
		GEngine->AddOnScreenDebugMessage(-1, 0.0f, MyColor, FString::Printf(TEXT("Sound Name= %s"), *str));
	}
	else
	{
		GEngine->AddOnScreenDebugMessage(-1, 0.0f, FColor::Red, FString::Printf(TEXT("Sound Name error= %i"), Result));
	}
}

This is the output

This is how it looks like in Studio:

[Leah_FMOD]

@apfelbaum

In the end I want this to be in Unreal Engine 5.2.

It is entirely possible to create a DSP plugin solely in Unreal, but that also entails manually placing it in the DSP chain and managing its lifetime yourself - you cannot set it up in your Studio project, and you cannot rely on Studio to manage it.

As for not being able to find the DSP itself, your code seems fine at a glance. I would recommend connecting to your project using the Core API profiler, which will let you visualize the DSP chain and identify whether your plugin is actually a part of the DSP chain. You can find the Core API profiler in your FMOD Studio API install’s bin folder. For me on Windows, that’s C:\Program Files (x86)\FMOD SoundSystem\FMOD Studio API Windows\bin\FMOD Profiler.exe.

[apfelbaum]

This tool is cool! it is Engine_20. I cannot see my plugin, i guess because it counts as a sound? It has no input buffer, only a single output buffer.

Here is the overview:

In Studio (part 1):

In Studio (part 2):

Edit.: I cannot find out but it it possible to get a track/channel deterministic? When I know that I can grab channel GIN_Accel by name or the order of the channels in the graph I can simply set the channel frequency like this once. I can workaround the rest manually but this is the last task I have. Everything else works fine so far.

[Leah_FMOD]

As per the section of our DSP White Paper that details common DSP units, DSP that are played on a channel consist of a sub-network of a fader unit and a resampler unit. I can see at least one resampler unit in your DSP graph, but if you wish to verify that the DSP is your plugin, you can right click on the node in the graph and deactivate/bypass it and observe the result.

As for getting the ChannelGroup/track order, this is not solely possible in the API. There are two possible ways to go figuring out which ChannelGroups are playing your plugin:

• Traverse the DSP graph in UE until you find the resampler units that are usages of your plugin (assuming you have no other plugin instruments in the event)
• In Studio, calculate the order of ChannelGroups retrieved by ChannelGroup::getGroup, write that info to the EventDescription’s user property, and retrieve and parse that info with Studio::EventDescription::getUserProperty

If the latter is more applicable to you, then please see the following reply for a Studio script that will write the order of Audio Tracks to the Event’s description: How can I get each Tracks Volume Level? - #9 by Louis_FMOD

[apfelbaum]

Somehow I cannot find the plugin like this so I created a 2nd plugin which simply gives me a control knob to set the frequency. I can simply query this plugin in code (meaybe because it has a in- and ouputbuffer?) and set the frequency of a channel in code once where this plugin is found. I afterwards simply return the update function to “Dont Process”.

This works in game but obviously this doesnt work in FMOD so I have to blindly adjust parameters in FMOD to compare the sound ingame. Is there the possiblity to set a channel frequency manually in FMOD?
If not, could you expose that, at default it is set to “auto” and you can manually change the samplerate to either a custom value or a selection of possible values, like Hz values in 1000Hz steps or so.

Edit.: If I have to write another plugin just to set the frequency of a channel in FMOD this is no problem at all, as long it is possible. I am already digging through the docs rn.

[apfelbaum]

Ok so I have another idea.
My issue currently is this: I pair the granular synth with looped samples. However the sound is always slightly out of tune, the reason i don’t know yet but I have an assumption (besides the phase): My granular synth does not output the exact sound at the input rpm but a value nearby (it does not pitch the output samples). This works great so far as I have enough samples at every few rpm steps. My assumption for the slight detune is that my looped samples (which use a root pitch) are pitched by the input rpm but the granular synth is always off obviously.
My question now is, can I return the calculated rpm from the granular synth (at which it outputs the final samples) and feed this into the looped samples track as input value somehow?

Best regards

[Leah_FMOD]

@apfelbaum

Somehow I cannot find the plugin like this so I created a 2nd plugin which simply gives me a control knob to set the frequency. I can simply query this plugin in code (meaybe because it has a in- and ouputbuffer?) and set the frequency of a channel in code once where this plugin is found. I afterwards simply return the update function to “Dont Process”.

This works in game but obviously this doesnt work in FMOD so I have to blindly adjust parameters in FMOD to compare the sound ingame. Is there the possiblity to set a channel frequency manually in FMOD?
If not, could you expose that, at default it is set to “auto” and you can manually change the samplerate to either a custom value or a selection of possible values, like Hz values in 1000Hz steps or so.

While this is possible in the API, it isn’t possible in the FMOD Studio application, as there’s no way to retrieve the handle of the FMOD System from within the DSP itself. It wouldn’t be possible even if you created another plugin solely for that purpose. Implementing your own resampling code in the plugin itself would be the simplest course of action.

@apfelbaum

My issue currently is this: I pair the granular synth with looped samples. However the sound is always slightly out of tune, the reason i don’t know yet but I have an assumption (besides the phase): My granular synth does not output the exact sound at the input rpm but a value nearby (it does not pitch the output samples). This works great so far as I have enough samples at every few rpm steps. My assumption for the slight detune is that my looped samples (which use a root pitch) are pitched by the input rpm but the granular synth is always off obviously.

This isn’t an issue with phase - it’s likely to be an issue with the samplerate. I cannot really help beyond that, as I’m not familiar with your plugin.

@apfelbaum

My question now is, can I return the calculated rpm from the granular synth (at which it outputs the final samples) and feed this into the looped samples track as input value somehow?

As with before, this may be possible in the API, but will not be possible in the FMOD Studio application. Again, I would recommend implementing your own resampling in your plugin.

[apfelbaum]

Okay so different question: Looking at the examples there are a few plugins like “generate_tone” which completely setup channel, plugins or sounds at startup of the application up to the main loop where everything is updated. Is there some kind of way to do this in unreal? A callback of an event which is called maybe on init and on update so I can get and set the parameters in Unreal Engine.