Lipsync when instance volume was set to 0 on the VCA

[aleveque]

Hello,

I tried to use SALSA to do lipsync.
For this, I rewired things and it usually work pretty good. Usually.

..Because when the user wants to set the volume to 0, then the system doesn’t get data (or a flat line).

The event is wired to a VCA, whose volume has been set to 0
The instance is retrieved by the lipsync system which creates a DSP to its channel group.

Here’s the code:

private void OnVoicePlaying(EventInstance eventInstance)
{
	if (m_playingInstances == null)
		return;

	if (!ShouldHaveSalsaUponInit)
		this.m_salsa = this.GetSalsa();

	if (!this.m_salsa)
		return;

	/*if (stopRoutine != null)
		StopCoroutine(stopRoutine);
	stopRoutine = null;*/

	// Define a basic DSP that receives a callback each mix to capture audio
	FMOD.DSP_DESCRIPTION desc = new FMOD.DSP_DESCRIPTION();
	desc.numinputbuffers = 1;
	desc.numoutputbuffers = 1;
	desc.read = m_readCallback;
	desc.userdata = GCHandle.ToIntPtr(m_objHandle);

	// Create an instance of the capture DSP and attach it to the master channel group to capture all audio
	FMOD.ChannelGroup eventCG;
	FMOD.DSP captureDSP;

	FMOD.RESULT result;

	result = eventInstance.getChannelGroup(out eventCG);
	if (VerboseFmodResults)
		Debug.Log("getChannelGroup => " + result);

	result = FMODUnity.RuntimeManager.CoreSystem.createDSP(ref desc, out captureDSP);
	if (VerboseFmodResults)
		Debug.Log("createDSP => " + result);

	result = captureDSP.setChannelFormat(FMOD.CHANNELMASK.MONO, 1, FMOD.SPEAKERMODE.MONO);
	if (VerboseFmodResults)
		Debug.Log("setChannelFormat => " + result);

	result = eventCG.addDSP(0, captureDSP);
	if (VerboseFmodResults)
		Debug.Log("addDSP => " + result);

	m_isPlaying = true;

	m_salsa.getExternalAnalysis = null;
	m_salsa.getTriggerIndex = null;
	m_salsa.InitializeDelegates(); // ensures those two delegates to be at their init values.

	m_salsa_getTriggerIndex = m_salsa.getTriggerIndex;  // gets the previous delegate in our own ref.
	m_salsa.getTriggerIndex = GetTriggerIndexForSalsa;	// ..So that it would be called within our delegate override

	m_salsa.getExternalAnalysis = GetAnalysisForSalsa;
	
	m_playingInstances.Add(eventInstance, captureDSP);
}

How can I wire the DSP so it can get the data before the VCA fades the volume to 0?

Is there a tool that we could use to visualise the ChannelGroup ?
On this page: https://www.fmod.com/docs/2.02/api/using-dsp-effects-in-the-core-api.html
It feels like a tree

[aleveque]

I just created this:

#if UNITY_EDITOR
private void Editor_VisualiseChannelGroup(StringBuilder str, int tabs, FMOD.ChannelGroup eventCG)
{
	FMOD.RESULT retval;

	string tabsStr = new string('\t', tabs);

	string name;
	retval = eventCG.getName(out name, 255);
	if (retval != FMOD.RESULT.OK)
	{
		str.Append(tabsStr).AppendLine("<color=red>ERROR getName</color>");
	}

	str.Append(tabsStr).Append(name).Append(" (").Append(eventCG.handle).AppendLine(")");

	int numDSP;
	retval = eventCG.getNumDSPs(out numDSP);
	if(retval != FMOD.RESULT.OK)
	{
		str.Append(tabsStr).Append("\t<color=red>ERROR getNumDSPs</color>");
	}
	else
	{
		FMOD.DSP dsp;
		FMOD.DSP subDsp;
		FMOD.DSPConnection dspCo;
		uint version;
		int channels, w, h;
		int numIn, numOut;

		for (int i = 0; i < numDSP; i++)
		{
			retval = eventCG.getDSP(i, out dsp);
			if (retval != FMOD.RESULT.OK)
			{
				str.Append(tabsStr).Append("\t<color=red>ERROR getDSP(").Append(i).AppendLine("</color>");
			}
			else
			{
				retval = dsp.getInfo(out name, out version, out channels, out w, out h);
				if (retval != FMOD.RESULT.OK)
				{
					str.Append(tabsStr).Append("\t<color=red>ERROR DSP::getInfo(").Append(i).AppendLine("</color>");
				}
				else
				{
					str.Append(tabsStr).Append("\t- DSP: ").Append(name).Append(" (").Append(dsp.handle).AppendLine(")");

					retval = dsp.getNumInputs(out numIn);
					if (retval != FMOD.RESULT.OK)
					{
						str.Append(tabsStr).Append("\t\t<color=red>ERROR DSP::getNumInputs(").Append(i).AppendLine("</color>");
					}
					else
					{
						for (int d = 0; d < numIn; d++)
						{
							retval = dsp.getInput(d, out subDsp, out dspCo);
							if (retval != FMOD.RESULT.OK)
							{
								str.Append(tabsStr).Append("\t\t<color=red>ERROR DSP::getInput(").Append(i).AppendLine("</color>");
							}
							else
							{
								retval = subDsp.getInfo(out name, out version, out channels, out w, out h);
								if(retval == FMOD.RESULT.OK)
									str.Append(tabsStr).Append("\t\t- in: ").Append(name).Append(" (").Append(subDsp.handle).AppendLine(")");
								else
									str.Append(tabsStr).Append("\t\t- in: ").Append(subDsp.handle).AppendLine();
							}
						}
					}

					retval = dsp.getNumOutputs(out numOut);
					if (retval != FMOD.RESULT.OK)
					{
						str.Append(tabsStr).Append("\t\t<color=red>ERROR DSP::getNumOutputs(").Append(i).AppendLine("</color>");
					}
					else
					{
						for (int d = 0; d < numOut; d++)
						{
							retval = dsp.getOutput(d, out subDsp, out dspCo);
							if (retval != FMOD.RESULT.OK)
							{
								str.Append(tabsStr).Append("\t\t<color=red>ERROR DSP::getOutput(").Append(i).AppendLine("</color>");
							}
							else
							{
								retval = subDsp.getInfo(out name, out version, out channels, out w, out h);
								if (retval == FMOD.RESULT.OK)
									str.Append(tabsStr).Append("\t\t- out: ").Append(name).Append(" (").Append(subDsp.handle).AppendLine(")");
								else
									str.Append(tabsStr).Append("\t\t- out: ").Append(subDsp.handle).AppendLine();
							}
						}
					}
				}
			}
		}
	}


	int numCG;
	retval = eventCG.getNumGroups(out numCG);
	if (retval != FMOD.RESULT.OK)
	{
		str.Append(tabsStr).Append('\t').AppendLine("<color=red>ERROR getNumGroups</color>");
	}
	else
	{
		FMOD.ChannelGroup subCG;
		for (int i = 0; i < numCG; i++)
		{
			retval = eventCG.getGroup(i, out subCG);
			if (retval != FMOD.RESULT.OK)
			{
				str.Append(tabsStr).Append('\t').Append("<color=red>ERROR getGroup(").Append(i).AppendLine("</color>");
			}
			else
			{
				Editor_VisualiseChannelGroup(str, tabs + 1, subCG);
			}
		}
	}
}
#endif

It showed me the DSP chain.

I added desc.name = Encoding.ASCII.GetBytes("lipsync_capture");

The thing is my DSP was at the end of it, so I changed the code at the “addDSP” :

int numDSP;

result = eventCG.getNumDSPs(out numDSP);
if (VerboseFmodResults)
Debug.Log("getNumDSPs => " + result);

result = eventCG.addDSP(numDSP-1, captureDSP);
if (VerboseFmodResults)
Debug.Log("addDSP => " + result);

Now the DSP seems better placed, but it still seem to be influenced by the VCA.

How can I prevent that?

[aleveque]

If this can help, here is what showed the method:

Visualizing channel group:
Master Bus (1589916096008)
    - DSP: FMOD Limiter (1585957525760)
        - in: FMOD Send (1586331779360)
        - out: ChanGroup Fader (1586343061776)
    - DSP: FMOD Send (1586331779360)
        - in: FMOD 3-EQ (1581531999824)
        - out: FMOD Limiter (1585957525760)
    - DSP: FMOD 3-EQ (1581531999824)
        - in: FMOD Pan (1590477095776)
        - out: FMOD Send (1586331779360)
    - DSP: FMOD Pan (1590477095776)
        - in: FMOD Fader (1586343052944)
        - out: FMOD 3-EQ (1581531999824)
    - DSP: FMOD Fader (1586343052944)
        - in: FMOD Pan (1590477099472)
        - out: FMOD Pan (1590477095776)
    - DSP: FMOD Pan (1590477099472)
        - in: lipsync_capture (1573133052288)
        - out: FMOD Fader (1586343052944)
    - DSP: lipsync_capture (1573133052288)
        - in: ChanGroup Fader (1586343076496)
        - out: FMOD Pan (1590477099472)
    - DSP: ChanGroup Fader (1586343076496)
        - in: ChanGroup Fader (1586343066192)
        - in: ChanGroup Fader (1586343072816)
        - in: ChanGroup Fader (1586343057360)
        - in: ChanGroup Fader (1586343083856)
        - in: ChanGroup Fader (1586343069136)
        - in: ChanGroup Fader (1586343069872)
        - in: ChanGroup Fader (1586343079440)
        - out: lipsync_capture (1573133052288)
    Group Bus (1589916115464)
        - DSP: ChanGroup Fader (1586343066192)
            - out: ChanGroup Fader (1586343076496)
    Group Bus (1589916092360)
        - DSP: ChanGroup Fader (1586343072816)
            - out: ChanGroup Fader (1586343076496)
    Group Bus (1589916092968)
        - DSP: ChanGroup Fader (1586343057360)
            - out: ChanGroup Fader (1586343076496)
    Group Bus (1589916093576)
        - DSP: ChanGroup Fader (1586343083856)
            - out: ChanGroup Fader (1586343076496)
    Group Bus (1589916108168)
        - DSP: ChanGroup Fader (1586343069136)
            - in: Channel Fader (1586297345952)
            - out: ChanGroup Fader (1586343076496)
    Group Bus (1589916094184)
        - DSP: ChanGroup Fader (1586343069872)
            - out: ChanGroup Fader (1586343076496)
    Group Bus (1589916098440)
        - DSP: ChanGroup Fader (1586343079440)
            - out: ChanGroup Fader (1586343076496)

[aleveque]

I made a method to visualize from the banks buses.

private void Editor_VisualiseBuses(StringBuilder str)
{
	FMOD.RESULT retval;

	Bank[] banks;
	retval = RuntimeManager.StudioSystem.getBankList(out banks);
	if(retval != FMOD.RESULT.OK)
	{
		str.AppendLine("<color=red>error for getBankList</color>");
		return;
	}

	string path;
	FMOD.Studio.Bus[] buses;

	string busPath;
	FMOD.ChannelGroup busCG;

	foreach (Bank b in banks)
	{
		retval = b.getPath(out path);
		if (retval != FMOD.RESULT.OK)
		{
			str.AppendLine("<color=red>error for getPath</color>");
			continue;
		}

		str.Append("Loaded bank: ").AppendLine(path);

		retval = b.getBusList(out buses);
		if (retval != FMOD.RESULT.OK)
		{
			str.AppendLine("<color=red>error for getBusList</color>");
			continue;
		}

		foreach (Bus bus in buses)
		{
			retval = bus.getPath(out busPath);
			if (retval != FMOD.RESULT.OK)
			{
				str.AppendLine("\t<color=red>error for Bus::getPath</color>");
				continue;
			}

			str.Append("\t- Found bus: ").AppendLine(busPath);

			retval = bus.getChannelGroup(out busCG);
			if (retval != FMOD.RESULT.OK)
			{
				str.AppendLine("\t\t<color=red>error for Bus::getChannelGroup</color>");
				continue;
			}

			Editor_VisualiseChannelGroup(str, 2, busCG);
		}
	}

}

Found out that the volume has been sent to 0 at the “Master bus” parent’s parent

I have like:

The bus “bus:/” → CG “Master Bus” → CG “Group Bus” whose volume is 0 (I guess the VCA integrates here) → CG “Input Bus” → CG “Master Bus” (Master of my event) → the DSP “lipsync_capture”

[aleveque]

Clamping the volume to a minimum of 0.001 seems to be working. My guess is that volume 0 means muted and there might be some optimization in the process.

I’m still curious about which idea you might have to have this nicely done ^^

• Sylafrs.

[Leah_FMOD]

Apologies for the delayed response!

@aleveque

My guess is that volume 0 means muted and there might be some optimization in the process.

Your guess is correct - what you’re observing is FMOD’s virtual voice system, which is always enabled when using the Studio API. When any particular channel or ChannelGroup produces no output, FMOD will virtualize them, meaning that while the playback position and any relevant properties will continue to be tracked, sample data will not be processed. When the VCA attenuates the event volume to 0, the event functionally produces no audio output and is virtualized away, so the sample data you’re trying to access isn’t present in the buffer.

Clamping the volume slightly above 0 is indeed one way to address the issue, as the event will continue to process and produce (inaudible) output, which you can access via your DSP. You can also use FMOD’s priority controls: the “Priority” event macro when set to “Highest”, and the Channel priority setting when set to 0 with Channel.setPriority(), will respectively cause the event/channel to never virtualize, even when no output is produced.