when i extract specturm from fmod the code is:
RuntimeManager.CoreSystem.createDSPByType(FMOD.DSP_TYPE.FFT, out m_FFTDsp);
RuntimeManager.CoreSystem.getMasterChannelGroup(out master);
var m_Result = master.addDSP(FMOD.CHANNELCONTROL_DSP_INDEX.HEAD, m_FFTDsp);
m_FFTDsp.setMeteringEnabled(true, true);
the lip sync is right.
but when i change the volume, the lip action disappear.
does volume affect spectrum.
extract spectrum code like this:
public string RecognizeByAudioSource(FMOD.DSP m_FFTDsp, int rate)
{
string result = null;
IntPtr unmanagedData;
uint length;
m_FFTDsp.getParameterData((int)FMOD.DSP_FFT.SPECTRUMDATA, out unmanagedData, out length);
FMOD.DSP_PARAMETER_FFT fftData = (FMOD.DSP_PARAMETER_FFT)Marshal.PtrToStructure(unmanagedData, typeof(FMOD.DSP_PARAMETER_FFT));
if (fftData.spectrum != null && fftData.spectrum.Length > 0)
{
playingAudioSpectrum = fftData.spectrum[0];
Recognize(ref result, rate);
}
return result;
}
I might need a little more context for your project to understand the problem here, what do you mean by lip sync?
Yes, if you turn the volume to 0 there will be no spectrum data, and if you turn the volume up too high you will get clipping, which will reflect in the spectrum data.
i want to use spectrum data to control person’s lip action.
i extract vowel word to control the bone of lip.
the bone weight of a, e, i, o, u extracted from the analysis of spectrum.
so, if i turn the volume to zero, the specturm will be equal to zero, then i can not extract vowel word.
when the player turn the bmg volum to zero, the role’s dialog lip action will disappear. it will affect the game experience.
do u have other workaround?
Thank you for the additional information, I can see what you mean now. I think you will need to add a Fader DSP after your FFT DSP, and then control the volume on the fader using setParameterFloat with the FMOD_DSP_FADER_GAIN parameter as the index argument. Then you can hook that up to the player’s volume controls instead of a group/VCA and that should still allow the FFT to get data regardless of what the fader volume is set to.
In this code snippet you are not calling ChannelGroup.addDSP on master, changing it to this is working for me:
public void InitFaderDSP()
{
if (m_addedFaderDSP) return;
m_addedFaderDSP = true;
var result = RuntimeManager.CoreSystem.createDSPByType(FMOD.DSP_TYPE.FADER, out m_faderDsp);
result = RuntimeManager.CoreSystem.getMasterChannelGroup(out master);
result = master.addDSP(-1, m_faderDsp); // Add DSP to master group
}
sorry, i forgot to addDSP to master group.
however, after i added the m_faderDsp to master group,
m_faderDsp.setParameterFloat((int)FMOD.DSP_FADER.GAIN, volume);
still can not control volume.
will the index -1 affect the fader dsp.
master.addDSP(-1, m_faderDsp);
i do not clearly understand the index, i can not find the rule of index when i try add several dsp to mater.
so trouble u again!
the full code is:
using FMODUnity;
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.UI;
public class FmodDsp : MonoBehaviour
{
public Slider m_slider;
private bool m_addedFaderDSP = false;
private FMOD.DSP m_faderDsp;
private FMOD.ChannelGroup master;
public void OnValueChange(float val)
{
var res = m_faderDsp.setParameterFloat((int)FMOD.DSP_FADER.GAIN, m_slider.value);
Debug.LogError(res + " volume=" + m_slider.value);
}
public void InitFaderDSP()
{
if (m_addedFaderDSP) return;
m_addedFaderDSP = true;
var result = RuntimeManager.CoreSystem.createDSPByType(FMOD.DSP_TYPE.FADER, out m_faderDsp);
result = RuntimeManager.CoreSystem.getMasterChannelGroup(out master);
result = master.addDSP(-1, m_faderDsp);
}
public void AddDSP()
{
InitFaderDSP();
}
}
1.first init dsp
2.and then dray the silder to control the volume. the rOnValueChange method is the callback of silder.
What are the min and max values on your slider? The FMOD.DSP_FADER.GAIN parameter value needs to be in the range of -80 to 10.
As for the -1 index, that coincides with placing the DSP at the head of the signal chain. You can read more about special locations here: FMOD_CHANNELCONTROL_DSP_INDEX. I believe you will need to place it at FMOD_CHANNELCONTROL_DSP_INDEX.TAIL (-3) for your use case.
thank u very much.
i change the min and max, it can control volume.
but my lip action still be affected by the volume.
i put the fader dsp to -3 index, and fft dsp to -2 dsp as the picture above.
so, what is the problem will be about?
i change the index of fft dsp to -2, and fader dsp index to -1.
then the volume will not affect the specturm data.
but i have a question, the index of dsp can be -3, -2, -1
-3 and -1, which one is the head.
i think fader dsp should be after the fft dsp.
but from the result, we i set fft dsp index to -2, and fader dsp to -1, the reuslt is right.
this seems -1 the tail.
so can u explain the reason for me.
I recommend you read the DSP White Paper which will explain everything in detail.
The problem is that the VCA is sitting before the FFT in your signal chain, so cutting off any audio from the VCA cuts off any audio that gets to your FFT. You will need to attach the FFT before the VO group. It is a little tricky attaching a dsp to a group, the easiest way would be to play an event that is on the VO group, then get it’s parent group, which should be the VO group. Then you can attach the FFT there. Here is a basic example just to illustrate what I’m talking about:
void Start()
{
StartCoroutine(AddDSPAsync());
}
IEnumerator AddDSPAsync()
{
FMOD.RESULT result;
result = RuntimeManager.StudioSystem.getEvent("event:/VO/Welcome", out FMOD.Studio.EventDescription ed);
result = ed.createInstance(out FMOD.Studio.EventInstance instance);
result = instance.start();
FMOD.ChannelGroup cg;
while(instance.getChannelGroup(out cg) != FMOD.RESULT.OK)
{
yield return null;
}
result = cg.getParentGroup(out FMOD.ChannelGroup vo);
result = RuntimeManager.CoreSystem.createDSPByType(FMOD.DSP_TYPE.FFT, out FMOD.DSP fft);
result = vo.addDSP(-1, fft);
result = instance.stop(0);
result = instance.release();
}
for example, i have two events A and B, there are both on tone vca.
i play two events.
i get event A channel group A_C, and event b channel group B_C.
the parent group of event A is A_PC, the parent group of event B is B_PC,
but the hashcode of A_PC and B_PC is not the same.
so, if i to add a fft dsp to A_PC, it can not affect B_PC.
so why?
can u tell me, why fader dsp’s value is at range -80 to 10 from your early response.
and i found the volume decreasion is not linear. the volume disappear at about -40.
so what is the decreasion curve of the fader dsp.
i want to remap the value [-80,10] to the linear control volume, just like vca set volume. vca control volume is linear.
i have a group of events, how to get the root parent of the these events.
u mean i just play one of it, and get the parent of this event?
but how many times getParentGroup will be called?
var result = emiters[0].EventInstance.getChannelGroup(out FMOD.ChannelGroup g1);
result = g1.getParentGroup(out FMOD.ChannelGroup parentGroup);
result = parentGroup.getParentGroup(out FMOD.ChannelGroup parentGroup2);
result = parentGroup2.getParentGroup(out FMOD.ChannelGroup parentGroup3);
toneGroup = parentGroup3;
result = toneGroup.addDSP(-1, m_faderDsp);
in the above code , i find three times. is there a easier way to get the root parent of tone group.
