Creating a custom plugin

apfelbaum · February 13, 2023, 11:00am

Hi,
I just started creating my own plugin. I guess I have the includes and dll. linking for the fmod studio apu correctly setup. What I want to do now is create a granular synth on my own.
The general idea is to have a table which describes the sample in an audio file (num samples, sample length etc.). All the logic to calculate the correct samples is already done given an input parameter (spoiler alert, it is a GS for car engine sounds so the input parameter is the RPM value).
For this I need to directly set the samples that FMOD should play with additional pitching. Is that possible in FMOD and where should I start? I have a bit trouble reading just your docs and looking at the example plugins.

Edit.: Idea is to put the plugin like the AudioMotors plugin into the timeline and adjust the RPM parameter with curves. The audio file can be drag and dropped onto the plugin. It can be whatever FMOD supports. The arrays of data could pasted manually or saved in the header of the audio file.

Best regards

Leah_FMOD · February 14, 2023, 5:24am

A custom DSP plugin will allow you to accomplish what you want - If your DSP has zero input channels, it is treated as an instrument, and can be placed on a sheet in an event. If you specify that your DSP has a data parameter of type DSP_PARAMETER_DATA_TYPE_USER, the default Studio UI component for a data parameter will allow you to browse for or drag and drop an audio file into the DSP effect. I’d recommend taking a look at the Plugin Reference documentation for more info, and the API examples in your FMOD Studio API install, specifically the “fmod_gain” and “fmod_noise” examples.

I’m a little unclear as to what you’re trying to do with regards to the table of sample info and additional pitching that you’ve described - would you mind elaborating on exactly what you mean?

apfelbaum · February 14, 2023, 10:48am

Hi,
thank you for your answer. So the idea is to have a 2d array with the audio file in it where you access the first layer with the rpm and the second layer contains the audio samples at this position (the grain). With some calculations these samples are pulled out and returned to be played by FMOD. As the array of grains won’t contain all possible rpm values the pulled out grains are pitched slightly to match the rpm. Say you have an input value of 5100 rpm but only grains at 5050 and 5200. then you pull out the grains at 5050 and pitch them a bit up. This calculation is also already done.

Leah_FMOD · February 15, 2023, 11:57pm

That makes sense, thanks for explaining in more detail.

As an aside, I’ve done a little more research into custom DSP UI components, specifically the ability to make use of default instrument UI components (i.e. playlist) when creating a custom DSP instrument, which would likely make adding samples to your DSP more convenient. However, after discussion with one of our programmers, we’ve come to the conclusion that this functionality isn’t exposed, so I’ve added it to our internal feature/improvement tracker.

That said, the default UI component for the user data parameter, DataDrop, should be fine for your needs. If you’re not using too many individual samples, then you could just use an arbitrary amount of data parameters to assign samples - though obviously with too many samples this would cause some UI clutter. If you’re planning to have a single larger audio file with table data saved into the header, as you’ve stated that you could do, a single data parameter that you use for the whole file would be fine, as you could seek/chop the whole file as needed in your custom plugin .dll.

apfelbaum · June 19, 2023, 10:41am

So I think I got the DataDrop working. But how do I get the data so I can read the binary? Inside the header of the file that I will drop I have all necessary data to define the sample rate, num samples etc. and afterwards I have a huge array of samples.

FMOD_RESULT F_CALLBACK Ginsu_Synth_DspSetParamData(FMOD_DSP_STATE* dsp, int index, void* data, unsigned int length)
{
	GinsuSynth* state = (GinsuSynth*)dsp->plugindata;
	FMOD_DSP_LOG(dsp, FMOD_DEBUG_LEVEL_ERROR, "Ginsu_Synth_DspSetParamData", "Text", "");



	return FMOD_ERR_INVALID_PARAM;
}

Leah_FMOD · June 20, 2023, 3:17am

When using FMOD_DSP_SETPARAM_DATA_CALLBACK, the void pointer data argument is a pointer to the binary data that is passed to the DSP (in this case via DataDrop). The length of the binary data is specified by the length argument. You’ll want to cast data to the appropriate type so you can access and operate on it as desired.

apfelbaum · June 20, 2023, 3:53pm

Okay I have a little problem with the .plugin.js. The Datadrop is not visible and the frequency randomization.

this is the result:
grafik

this is the script:

studio.plugins.registerPluginDescription("Ginsu Synth", {
    companyName: "Custom",
    productName: "Ginsu Synthesizer",

    deckUi: {
        deckWidgetType: studio.ui.deckWidgetType.Layout,
        layout: studio.ui.layoutType.HBoxLayout,
        spacing: 18,
        items: [
            {
                deckWidgetType: studio.ui.deckWidgetType.Layout,
                layout: studio.ui.layoutType.VBoxLayout,
                spacing: 12,
                items: [
                    { deckWidgetType: studio.ui.deckWidgetType.Pixmap, filePath: __dirname + "/ginsu_synth_logo.png" },
                    { deckWidgetType: studio.ui.deckWidgetType.DataDrop, binding: "Ginsu Synth Data", fileNameFilters: "*.gin" },
                ],
            },
            {
                deckWidgetType: studio.ui.deckWidgetType.Layout,
                layout: studio.ui.layoutType.GridLayout,
                spacing: 12,
                items: [
                    { deckWidgetType: studio.ui.deckWidgetType.Dial, color: "#DF171A", row: 0, column: 0, binding: "Rpm", },
                    { deckWidgetType: studio.ui.deckWidgetType.Dial, color: "#FFFFFF", row: 1, column: 0, binding: "Idle Rpm", },
                    { deckWidgetType: studio.ui.deckWidgetType.Dial, color: "#DF171A", row: 1, column: 1, binding: "Redline Rpm", },
                    { deckWidgetType: studio.ui.deckWidgetType.Dial, color: "#DF171A", row: 1, column: 2, binding: "Frequency Randomization", },
                ],
            },
        ],
    },
});

and this is in code

extern "C"
{
	F_EXPORT FMOD_DSP_DESCRIPTION* F_CALL FMODGetDSPDescription()
	{
		FMOD_DSP_INIT_PARAMDESC_FLOAT(p_rpm, "Rpm", "rpm", "0 to 10.000. Default = 1000", GINSU_SYNTH_PARAM_RPM_MIN, GINSU_SYNTH_PARAM_RPM_MAX, GINSU_SYNTH_PARAM_RPM_DEFAULT);
		FMOD_DSP_INIT_PARAMDESC_FLOAT(p_idle_rpm, "Idle Rpm", "rpm", "0 to 5.000. Default = 1000", GINSU_SYNTH_PARAM_IDLE_RPM_MIN, GINSU_SYNTH_PARAM_IDLE_RPM_MAX, GINSU_SYNTH_PARAM_IDLE_RPM_DEFAULT);
		FMOD_DSP_INIT_PARAMDESC_FLOAT(p_redline_rpm, "Redline Rpm", "rpm", "5.000 to 10.000. Default = 10000", GINSU_SYNTH_PARAM_REDLINE_RPM_MIN, GINSU_SYNTH_PARAM_REDLINE_RPM_MAX, GINSU_SYNTH_PARAM_REDLINE_RPM_DEFAULT);
		FMOD_DSP_INIT_PARAMDESC_FLOAT(p_frequency_randomization, "Frequency Randomization", "rpm", "0 to 100. Default = 0", GINSU_SYNTH_PARAM_FREQUENCY_RANDOMIZATION_MIN, GINSU_SYNTH_PARAM_FREQUENCY_RANDOMIZATION_MAX, GINSU_SYNTH_PARAM_FREQUENCY_RANDOMIZATION_DEFAULT);
		FMOD_DSP_INIT_PARAMDESC_DATA(p_ginsu_synth_data, "Ginsu Synth Data", ".gin file", "Some description", GINSU_SYNTH_PARAM_DATATYPE_DEFAULT);
		return &Ginsu_Synth_Desc;
	}
}

when i don’t use the .js all controls are there but pretty ugly aligned.

Best regards.

Edit.: Found out, the name of the control shouldn’t be too long.

apfelbaum · June 22, 2023, 9:09pm

Okay so I have it now. The plugin is almost finished I think.

grafik

What I have here is the number of samples (which size is actually variable, i just gave it 512 slots for testing). The number of samples depends on the rpm of the car and is calculated. The samples are in int16_t and have a samplerate of 32khz. How do I get it into FMOD correctly?

Edit.:

Found out when I restructure the function like this I have output.

Do I have to change anything else? Is it correct that the buffer float are normalized to 1.0?

apfelbaum · June 23, 2023, 2:32pm

You can hear the first car sound being pitched too high because its sample rate is lower than fmods sample rate, the second sample rate is matching and sounds correct.

I fixed all bugs regarding the buffer. Now I just need to know how to set the samplerate for the plugin if that is possible. Or do I have to resample myself?

Leah_FMOD · June 25, 2023, 11:06pm

Happy to see you’ve managed to figure most of it out!

Unfortunately, there’s no way to get the FMOD system to resample the DSP output for you, so you will have to resample the output yourself.

apfelbaum · June 26, 2023, 7:45am

I guess you don’t have a function for resampling in built that you can expose so one can call it in a DSP?

Leah_FMOD · June 26, 2023, 9:55pm

Unfortunately, there isn’t one, though I’ve noted your interest on our internal feature/improvement tracker.

There is Channel::setFrequency which (essentially) adds a resampler to a channel, but that function isn’t available to DSPs. When using the API, you could use the function on a channel that the DSP is on, but that would involve passing in either the channel or entire system to the DSP as user data - something that isn’t possible in Studio, as you have no way to access a handle to the system or channel.

apfelbaum · June 27, 2023, 8:04pm

In the end I want this to be in Unreal Engine 5.2.

Btw., your help is super good and I love the support.

Edit.: I see by recursively looking for the channelgroups in Unreal Engine I am able to find all DSPs in channels. But somehow only “Channel Fader/ChanGroupFader/FMOD Pan” is found, not the Plugin instrument(s). Is there another way to find these?

void UGinsuSoundTest::TickComponent(float DeltaTime, enum ELevelTick TickType, FActorComponentTickFunction* ThisTickFunction)
{
	Super::TickComponent(DeltaTime, TickType, ThisTickFunction);

	if (AudioComp == nullptr) return;
	FMOD::Studio::EventInstance* Instance = AudioComp->StudioInstance;
	if (Instance == nullptr) return;

	FMOD_STUDIO_PLAYBACK_STATE PlaybackState;
	FMOD_RESULT Result = Instance->getPlaybackState(&PlaybackState);
	if (Result != FMOD_OK || PlaybackState != FMOD_STUDIO_PLAYBACK_PLAYING) return;

	FMOD::ChannelGroup* ChannelGroup;
	Result = Instance->getChannelGroup(&ChannelGroup);
	if (Result != FMOD_OK) return;

	ColorIndex = 0;
	ExtractGroup(ChannelGroup);
}

void UGinsuSoundTest::ExtractGroup(FMOD::ChannelGroup* Group)
{
	MyColor = FColor::MakeRandomSeededColor(ColorIndex++);
	int32 NumDSPs = 0;
	FMOD_RESULT Result = Group->getNumDSPs(&NumDSPs);

	if (Result == FMOD_OK)
	{
		for (int32 j = 0; j < NumDSPs; ++j)
		{
			FMOD::DSP* DSP;
			Result = Group->getDSP(j, &DSP);
			if (Result != FMOD_OK) continue;

			ExtractDSP(DSP);
		}
	}

	int32 numChannels = 0;
	Result = Group->getNumChannels(&numChannels);
	if (Result == FMOD_OK)
	{
		for (int32 i = 0; i < numChannels; ++i)
		{
			FMOD::Channel* Channel;
			Result = Group->getChannel(i, &Channel);
			if (Result != FMOD_OK) continue;

			ExtractChannel(Channel);
		}
	}

	int32 NumGroups = 0;
	Result = Group->getNumGroups(&NumGroups);
	if (Result == FMOD_OK)
	{
		for (int32 i = 0; i < NumGroups; ++i)
		{
			FMOD::ChannelGroup* ChildGroup;
			Result = Group->getGroup(i, &ChildGroup);
			if (Result != FMOD_OK) continue;

			ExtractGroup(ChildGroup);
		}
	}
}

void UGinsuSoundTest::ExtractChannel(FMOD::Channel* Channel) const
{
	int32 numDSPs = 0;
	FMOD_RESULT Result = Channel->getNumDSPs(&numDSPs);
	if (Result == FMOD_OK)
	{
		for (int32 j = 0; j < numDSPs; ++j)
		{
			FMOD::DSP* DSP;
			Result = Channel->getDSP(j, &DSP);
			if (Result != FMOD_OK) continue;

			ExtractDSP(DSP);
		}
	}
	FMOD::Sound* Sound;
	Result = Channel->getCurrentSound(&Sound);
	if (Result == FMOD_OK)
	{
		ExtractSound(Sound);
	}

	float Frequency = -1.0f;
	Result = Channel->getFrequency(&Frequency);
	GEngine->AddOnScreenDebugMessage(-1, 0.0f, MyColor, FString::Printf(TEXT("Frequency= %fHz"), Frequency));
}

void UGinsuSoundTest::ExtractDSP(FMOD::DSP* DSP) const
{
	char name[50];
	uint32* Version = 0;
	int32 Channels = 0, ConfigWidth = 0, ConfigHeight = 0;
	FMOD_RESULT Result = DSP->getInfo(name, Version, &Channels, &ConfigWidth, &ConfigHeight);
	if (Result == FMOD_OK)
	{
		FString DSPName(name);
		GEngine->AddOnScreenDebugMessage(-1, 0.0f, MyColor, FString::Printf(TEXT("DSPName= %s, Channels= %i"), *DSPName, Channels));
	}
	else
	{
		GEngine->AddOnScreenDebugMessage(-1, 0.0f, FColor::Red, FString::Printf(TEXT("DSP found but without info")));
	}
}

void UGinsuSoundTest::ExtractSound(FMOD::Sound* Sound) const
{
	constexpr int32 SoundNameLength = 32;
	char SoundName[SoundNameLength];
	FMOD_RESULT Result = Sound->getName(SoundName, SoundNameLength);
	if (Result == FMOD_OK)
	{
		FString str = GetNullTerminatedString(SoundName, SoundNameLength);
		GEngine->AddOnScreenDebugMessage(-1, 0.0f, MyColor, FString::Printf(TEXT("Sound Name= %s"), *str));
	}
	else
	{
		GEngine->AddOnScreenDebugMessage(-1, 0.0f, FColor::Red, FString::Printf(TEXT("Sound Name error= %i"), Result));
	}
}

This is the output

grafik

This is how it looks like in Studio:

Leah_FMOD · June 29, 2023, 5:33am

It is entirely possible to create a DSP plugin solely in Unreal, but that also entails manually placing it in the DSP chain and managing its lifetime yourself - you cannot set it up in your Studio project, and you cannot rely on Studio to manage it.

As for not being able to find the DSP itself, your code seems fine at a glance. I would recommend connecting to your project using the Core API profiler, which will let you visualize the DSP chain and identify whether your plugin is actually a part of the DSP chain. You can find the Core API profiler in your FMOD Studio API install’s bin folder. For me on Windows, that’s C:\Program Files (x86)\FMOD SoundSystem\FMOD Studio API Windows\bin\FMOD Profiler.exe.

apfelbaum · June 29, 2023, 8:22am

This tool is cool! it is Engine_20. I cannot see my plugin, i guess because it counts as a sound? It has no input buffer, only a single output buffer.

Here is the overview:

In Studio (part 1):

In Studio (part 2):

Edit.: I cannot find out but it it possible to get a track/channel deterministic? When I know that I can grab channel GIN_Accel by name or the order of the channels in the graph I can simply set the channel frequency like this once. I can workaround the rest manually but this is the last task I have. Everything else works fine so far.

Leah_FMOD · June 29, 2023, 11:06pm

As per the section of our DSP White Paper that details common DSP units, DSP that are played on a channel consist of a sub-network of a fader unit and a resampler unit. I can see at least one resampler unit in your DSP graph, but if you wish to verify that the DSP is your plugin, you can right click on the node in the graph and deactivate/bypass it and observe the result.

As for getting the ChannelGroup/track order, this is not solely possible in the API. There are two possible ways to go figuring out which ChannelGroups are playing your plugin:

Traverse the DSP graph in UE until you find the resampler units that are usages of your plugin (assuming you have no other plugin instruments in the event)
In Studio, calculate the order of ChannelGroups retrieved by ChannelGroup::getGroup, write that info to the EventDescription’s user property, and retrieve and parse that info with Studio::EventDescription::getUserProperty

If the latter is more applicable to you, then please see the following reply for a Studio script that will write the order of Audio Tracks to the Event’s description: How can I get each Tracks Volume Level? - #9 by Leah_FMOD

apfelbaum · July 6, 2023, 11:42am

Somehow I cannot find the plugin like this so I created a 2nd plugin which simply gives me a control knob to set the frequency. I can simply query this plugin in code (meaybe because it has a in- and ouputbuffer?) and set the frequency of a channel in code once where this plugin is found. I afterwards simply return the update function to “Dont Process”.

This works in game but obviously this doesnt work in FMOD so I have to blindly adjust parameters in FMOD to compare the sound ingame. Is there the possiblity to set a channel frequency manually in FMOD?
If not, could you expose that, at default it is set to “auto” and you can manually change the samplerate to either a custom value or a selection of possible values, like Hz values in 1000Hz steps or so.

Edit.: If I have to write another plugin just to set the frequency of a channel in FMOD this is no problem at all, as long it is possible. I am already digging through the docs rn.

apfelbaum · July 9, 2023, 12:15pm

Ok so I have another idea.
My issue currently is this: I pair the granular synth with looped samples. However the sound is always slightly out of tune, the reason i don’t know yet but I have an assumption (besides the phase): My granular synth does not output the exact sound at the input rpm but a value nearby (it does not pitch the output samples). This works great so far as I have enough samples at every few rpm steps. My assumption for the slight detune is that my looped samples (which use a root pitch) are pitched by the input rpm but the granular synth is always off obviously.
My question now is, can I return the calculated rpm from the granular synth (at which it outputs the final samples) and feed this into the looped samples track as input value somehow?

Best regards

Leah_FMOD · July 14, 2023, 5:04am

While this is possible in the API, it isn’t possible in the FMOD Studio application, as there’s no way to retrieve the handle of the FMOD System from within the DSP itself. It wouldn’t be possible even if you created another plugin solely for that purpose. Implementing your own resampling code in the plugin itself would be the simplest course of action.

This isn’t an issue with phase - it’s likely to be an issue with the samplerate. I cannot really help beyond that, as I’m not familiar with your plugin.

As with before, this may be possible in the API, but will not be possible in the FMOD Studio application. Again, I would recommend implementing your own resampling in your plugin.

apfelbaum · July 16, 2023, 1:25pm

Okay so different question: Looking at the examples there are a few plugins like “generate_tone” which completely setup channel, plugins or sounds at startup of the application up to the main loop where everything is updated. Is there some kind of way to do this in unreal? A callback of an event which is called maybe on init and on update so I can get and set the parameters in Unreal Engine.

Topic		Replies	Views
Plugin in c++ - load a given set of audio files, loop it and modify it with FMOD DSP effects FMOD Engine cpp	3	506	March 28, 2023
Custom Plug-in instrument with audiofile loading FMOD Studio	1	29	December 4, 2024
How do i setup in Fmod Audiomotors effect to simulate an engine sound. wav file FMOD Studio	5	464	January 16, 2024
Granular Synthesis with FMOD Studio and Unity C# Unity unity , csharp	9	128	February 22, 2025
Custom DSP set data changes data for all instances FMOD Engine cpp	1	191	December 13, 2023

Creating a custom plugin

Related topics