How to input desktop audio so i can retrieve spectrum data from it WASAPI loopback capture

couchpotatochip21 · September 20, 2022, 4:52am

I want to input the left and right channels of desktop audio so i can retrieve seperate spectrum data of the lefft and right

jeff_fmod · September 25, 2022, 11:24pm

I recommend you take a look at the record_enumeration example, which can be found in the FMOD Engine download Core API example. That will show you how to record from a WASAPI loopback device and into a Sound object. Getting the spectrum data is then a matter of attaching an FFT DSP to the Sound’s Channel. We have a Spectrum Analysis example of how to achieve this in Unity.

couchpotatochip21 · September 27, 2022, 4:05am

I downloaded and imported the .unitypackage. i am struggling to find the examples you mentioned.

thank you for your previous response

jeff_fmod · September 27, 2022, 5:50am

No problem here are the steps in more detail:

Go to the FMOD Download page
Click on the tab labelled “FMOD Engine”
Download the FMOD Engine for Windows

FMOD Engine download1132×891 42.1 KB
Run the executable to install the FMOD Engine
Navigate to “C:\Program Files (x86)\FMOD SoundSystem\FMOD Studio API Windows”, this is the root directory of the FMOD Engine
Navigate down to “\api\core\examples\vs2019” and open examples.sln inside Visual studio 2019
Once that opens, right click on the record_enumeration project in the Solution Explorer and select Set as Startup Project

image596×705 32.6 KB
F5 to run

The code in the record_enumeration example will show you how to capture and playback from a loopback device. It is written in C++, which if you are familiar with C# should be understandable- just ignore any * or & and imagine that :: and -> are . operators, e.g

//C++
struct RECORD_STATE
{
    FMOD::Sound *sound;
    FMOD::Channel *channel;
};

//C#
struct RECORD_STATE
{
    FMOD.Sound sound;
    FMOD.Channel channel;
};

//C++
result = system->createSound(0, FMOD_LOOP_NORMAL | FMOD_OPENUSER, &exinfo, &record[cursor].sound);

//C#
result = system.createSound(0, FMOD_LOOP_NORMAL | FMOD_OPENUSER, exinfo, record[cursor].sound);

couchpotatochip21 · September 27, 2022, 3:49pm

Thank you so much, ill try it latere today

couchpotatochip21 · December 10, 2022, 5:04am

I put this off for a while

How should i implement this in unity?

Leah_FMOD · December 14, 2022, 5:01am

The basic steps to record and play from a WASAPI loopback are to:

Grab the rate and channel info of the loopback audio device you want audio from
Create an FMOD Sound using the retrieved info
Record from the loopback device into the FMOD Sound
Play the FMOD Sound

A very basic implementation in Unity might look something like this:

void Start()
{
    // grab the rate and channel info of the loopback audio device
    int deviceIndex = 0; // arbitrary index, pick whichever one corresponds to the device you want
    int nativeRate = 0;
    int nativeChannels = 0;
    result = FMODUnity.RuntimeManager.CoreSystem.getRecordDriverInfo(deviceIndex, out _, 0, out _, out nativeRate, out _, out nativeChannels, out _);

    // create extended sound info object using retrieved rate and channel info
    FMOD.CREATESOUNDEXINFO exinfo = new FMOD.CREATESOUNDEXINFO
    {
        cbsize = MarshalHelper.SizeOf(typeof(FMOD.CREATESOUNDEXINFO)),
        numchannels = nativeChannels,
        format = FMOD.SOUND_FORMAT.PCM16,
        defaultfrequency = nativeRate,
        length = (uint)(nativeRate * sizeof(short) * nativeChannels) 
    };

    // create a new FMOD Sound using the extended sound info,
    // and start recording from the device into that sound
    FMOD.Sound sound;
    result = FMODUnity.RuntimeManager.CoreSystem.createSound("", FMOD.MODE.LOOP_NORMAL | FMOD.MODE.OPENUSER, ref exinfo, out sound);
    result = FMODUnity.RuntimeManager.CoreSystem.recordStart(deviceIndex, sound, true);

    // play the FMOD Sound (in this case, on the master channel group)
    FMOD.Channel channel;
    FMOD.ChannelGroup mcg;
    result = FMODUnity.RuntimeManager.CoreSystem.getMasterChannelGroup(out mcg);
    result = FMODUnity.RuntimeManager.CoreSystem.playSound(sound, mcg, false, out channel);
}

In the Core API example previous mentioned by Jeff, the record_enumeration example demonstrates swapping between different audio devices, though in your case you may simply want to enumerate the devices and find the specific loopback device you want to use. You might also run into some audio crackling, which is caused by drifting between the drivers of the audio device and FMOD - the record example shows how to dynamically adjust the playback speed to compensate for the drift while keeping the same latency.

To get the spectrum data, you can attach an FFT DSP to the channel the sound is playing on, and retrieve the spectrum data from it. The Spectrum Analysis Scripting Example shows how to do this.

Hope that helps!

couchpotatochip21 · March 21, 2023, 4:18am

Alot of crap went down but im back at it
So i had to add var to this line

var result = FMODUnity.RuntimeManager.CoreSystem.getRecordDriverInfo(deviceIndex, out _, 0, out _, out nativeRate, out _, out nativeChannels, out _);

Because it was not previously declared. I am also having problems with MarshalHelper as it does not exist in the current context

cbsize = MarshalHelper.SizeOf(typeof(FMOD.CREATESOUNDEXINFO)),

Assets\thing1.cs(25,22): error CS0103: The name ‘MarshalHelper’ does not exist in the current context

Full code for reference:

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using System.Runtime.InteropServices;

public class thing1 : MonoBehaviour
{

// Update is called once per frame
void Update()
{
    
}
void Start()
{
    // grab the rate and channel info of the loopback audio device
    int deviceIndex = 0; // arbitrary index, pick whichever one corresponds to the device you want
    int nativeRate = 0;
    int nativeChannels = 0;
    var result = FMODUnity.RuntimeManager.CoreSystem.getRecordDriverInfo(deviceIndex, out _, 0, out _, out nativeRate, out _, out nativeChannels, out _);

    // create extended sound info object using retrieved rate and channel info
    FMOD.CREATESOUNDEXINFO exinfo = new FMOD.CREATESOUNDEXINFO
    {
        cbsize = MarshalHelper.SizeOf(typeof(FMOD.CREATESOUNDEXINFO)),
        numchannels = nativeChannels,
        format = FMOD.SOUND_FORMAT.PCM16,
        defaultfrequency = nativeRate,
        length = (uint)(nativeRate * sizeof(short) * nativeChannels)
    };

    // create a new FMOD Sound using the extended sound info,
    // and start recording from the device into that sound
    FMOD.Sound sound;
    result = FMODUnity.RuntimeManager.CoreSystem.createSound("", FMOD.MODE.LOOP_NORMAL | FMOD.MODE.OPENUSER, ref exinfo, out sound);
    result = FMODUnity.RuntimeManager.CoreSystem.recordStart(deviceIndex, sound, true);

    // play the FMOD Sound (in this case, on the master channel group)
    FMOD.Channel channel;
    FMOD.ChannelGroup mcg;
    result = FMODUnity.RuntimeManager.CoreSystem.getMasterChannelGroup(out mcg);
    result = FMODUnity.RuntimeManager.CoreSystem.playSound(sound, mcg, false, out channel);
}

}

unity version: 2021.3.6f1

Leah_FMOD · March 21, 2023, 10:28pm

MarshalHelper is a member of the FMOD namespace, so you either need to add using FMOD; to the top of your script, or specify that you’re using that namespace in that line with cbsize = FMOD.MarshalHelper.SizeOf(typeof(FMOD.CREATESOUNDEXINFO)),

couchpotatochip21 · March 25, 2023, 4:15am

Your solution worked Thanks!
Could you please tell me I split the two channels off? I need to spectrum the left and right seperately.

Leah_FMOD · March 27, 2023, 6:01am

Doing so will require a bit of fiddling with the DSP chain - the following code snippet, placed after the previous one, essentially duplicates the signal splitting into each separate channel, and routes them through two FFT DSPs. I’ve declared lFFT and rFFT as class members of type FMOD.DSP.

        // create FFT DSP for left channel
        result = FMODUnity.RuntimeManager.CoreSystem.createDSPByType(FMOD.DSP_TYPE.FFT, out lFFT);
        // add connection between sound channel fader and left FFT DSP
        result = lFFT.addInput(channelFader, out FMOD.DSPConnection lConnection, FMOD.DSPCONNECTION_TYPE.STANDARD);
        // set mix matrix on connection to remove right channel
        result = lConnection.setMixMatrix(new float[] { 1, 0 }, 1, 2);
        // set wet output of FFT DSP to 0, and set it to be active
        result = lFFT.setWetDryMix(1f, 0f, 0f);
        result = lFFT.setActive(true);
        // add out from DSP to master channel group fader
        result = mcgFader.addInput(lFFT, out _, FMOD.DSPCONNECTION_TYPE.STANDARD);

        // repeat the same for the right side, but swap the mix matrix so DSP only receives right channel
        result = FMODUnity.RuntimeManager.CoreSystem.createDSPByType(FMOD.DSP_TYPE.FFT, out rFFT);
        result = rFFT.addInput(channelFader, out FMOD.DSPConnection rConnection, FMOD.DSPCONNECTION_TYPE.STANDARD);
        result = rConnection.setMixMatrix(new float[] { 0, 1 }, 1, 2);
        result = rFFT.setWetDryMix(1f, 0f, 0f);
        result = rFFT.setActive(true);
        result = mcgFader.addInput(rFFT, out _, FMOD.DSPCONNECTION_TYPE.STANDARD);

From there, you can grab the spectrum data from each FFT with DSP::getParameterData. I would recommend reading over documentation for the functions used in the code snippet so that you have an understanding of what is being done.

An aside: since you’re interacting with the Core API, a lot of the safeguards and automatic management that the Studio API does is not available to you. As such, you’ll want to make sure that you don’t interfere with existing DSPs, and also to take care of any DSPs or sounds that you create yourself. You need to call release() on your sound and FFT DSPs after you’re done with them (e.g. in OnDestroy()) so that the Core System can handle cleaning them up, otherwise they will leak into memory. I’d recommend storing them all as class members to make this process easier.

couchpotatochip21 · April 4, 2023, 11:15pm

I believe i inserted the snippet wrong as I am getting a plethora of errors, primarily around IFFT.

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using FMOD;

public class thing1 : MonoBehaviour
{
    [SerializeField] int deviceIndex = 4; // arbitrary index, pick whichever one corresponds to the device you want (window's currently selected output is 1 btw, it will change to whatever the user sets their audio output to even if the program is currently running.)

    // Update is called once per frame
    void Update()
    {
        
    }
    void Start()
    {
        // grab the rate and channel info of the loopback audio device
        int nativeRate = 0;
        int nativeChannels = 0;
        var result = FMODUnity.RuntimeManager.CoreSystem.getRecordDriverInfo(deviceIndex, out _, 0, out _, out nativeRate, out _, out nativeChannels, out _);

        // create extended sound info object using retrieved rate and channel info
        FMOD.CREATESOUNDEXINFO exinfo = new FMOD.CREATESOUNDEXINFO
        {
            cbsize = MarshalHelper.SizeOf(typeof(FMOD.CREATESOUNDEXINFO)),
            numchannels = nativeChannels,
            format = FMOD.SOUND_FORMAT.PCM16,
            defaultfrequency = nativeRate,
            length = (uint)(nativeRate * sizeof(short) * nativeChannels)
        };

        // create a new FMOD Sound using the extended sound info,
        // and start recording from the device into that sound
        FMOD.Sound sound;
        result = FMODUnity.RuntimeManager.CoreSystem.createSound("", FMOD.MODE.LOOP_NORMAL | FMOD.MODE.OPENUSER, ref exinfo, out sound);
        result = FMODUnity.RuntimeManager.CoreSystem.recordStart(deviceIndex, sound, true);

        // play the FMOD Sound (in this case, on the master channel group)
        FMOD.Channel channel;
        FMOD.ChannelGroup mcg;
        result = FMODUnity.RuntimeManager.CoreSystem.getMasterChannelGroup(out mcg);
        result = FMODUnity.RuntimeManager.CoreSystem.playSound(sound, mcg, false, out channel);




        // create FFT DSP for left channel
        result = FMODUnity.RuntimeManager.CoreSystem.createDSPByType(FMOD.DSP_TYPE.FFT, out lFFT);
        // add connection between sound channel fader and left FFT DSP
        result = lFFT.addInput(channelFader, out FMOD.DSPConnection lConnection, FMOD.DSPCONNECTION_TYPE.STANDARD);
        // set mix matrix on connection to remove right channel
        result = lConnection.setMixMatrix(new float[] { 1, 0 }, 1, 2);
        // set wet output of FFT DSP to 0, and set it to be active
        result = lFFT.setWetDryMix(1f, 0f, 0f);
        result = lFFT.setActive(true);
        // add out from DSP to master channel group fader
        result = mcgFader.addInput(lFFT, out _, FMOD.DSPCONNECTION_TYPE.STANDARD);

        // repeat the same for the right side, but swap the mix matrix so DSP only receives right channel
        result = FMODUnity.RuntimeManager.CoreSystem.createDSPByType(FMOD.DSP_TYPE.FFT, out rFFT);
        result = rFFT.addInput(channelFader, out FMOD.DSPConnection rConnection, FMOD.DSPCONNECTION_TYPE.STANDARD);
        result = rConnection.setMixMatrix(new float[] { 0, 1 }, 1, 2);
        result = rFFT.setWetDryMix(1f, 0f, 0f);
        result = rFFT.setActive(true);
        result = mcgFader.addInput(rFFT, out _, FMOD.DSPCONNECTION_TYPE.STANDARD);
    }
    void OnDestroy()
    {
        release();
    }
}

Leah_FMOD · April 4, 2023, 11:43pm

In my snippet, lFFT and rFFT are declared as member variables of the class (i.e. thing1 in your code) that the code is contained in. As for mcgFader and channelFader, that’s my mistake - I forgot to include the relevant lines in my snippet. Add these lines above the just before snippet:

        // get master channel group fader
        FMOD.DSP mcgFader;
        result = mcg.getDSP(FMOD.CHANNELCONTROL_DSP_INDEX.FADER, out mcgFader);
        // get sound's fader
        FMOD.DSP channelFader;
        result = channel.getDSP(FMOD.CHANNELCONTROL_DSP_INDEX.HEAD, out channelFader);

Additionally, you need to call release on your sound and both FFT DSPs (i.e. sound.release(), lFFT.release(), etc.). Calling release(); as you’ve done tries to call a function of the class thing1 named “release”, which doesn’t exist.

couchpotatochip21 · July 10, 2023, 5:29am

I finally got around to trying things out and got it to work but not actually.
It isnt giving me the 20 errors from earlier so thank you
but it does give me this

any idea why?
thanks

couchpotatochip21 · July 10, 2023, 6:30pm

I forgot to put my code

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using FMOD;

public class thing1 : MonoBehaviour
{
    [SerializeField] int deviceIndex = 4; // arbitrary index, pick whichever one corresponds to the device you want (window's currently selected output is 1 btw, it will change to whatever the user sets their audio output to even if the program is currently running.)

    //added 7/9/2023


    // Update is called once per frame
    void Update()
    {
        
    }
    void Start()
    {
        // grab the rate and channel info of the loopback audio device
        int nativeRate = 0;
        int nativeChannels = 0;
        var result = FMODUnity.RuntimeManager.CoreSystem.getRecordDriverInfo(deviceIndex, out _, 0, out _, out nativeRate, out _, out nativeChannels, out _);

        // create extended sound info object using retrieved rate and channel info
        FMOD.CREATESOUNDEXINFO exinfo = new FMOD.CREATESOUNDEXINFO
        {
            cbsize = MarshalHelper.SizeOf(typeof(FMOD.CREATESOUNDEXINFO)),
            numchannels = nativeChannels,
            format = FMOD.SOUND_FORMAT.PCM16,
            defaultfrequency = nativeRate,
            length = (uint)(nativeRate * sizeof(short) * nativeChannels)
        };

        // create a new FMOD Sound using the extended sound info,
        // and start recording from the device into that sound
        FMOD.Sound sound;
        result = FMODUnity.RuntimeManager.CoreSystem.createSound("", FMOD.MODE.LOOP_NORMAL | FMOD.MODE.OPENUSER, ref exinfo, out sound);
        result = FMODUnity.RuntimeManager.CoreSystem.recordStart(deviceIndex, sound, true);

        // play the FMOD Sound (in this case, on the master channel group)
        FMOD.Channel channel;
        FMOD.ChannelGroup mcg;
        result = FMODUnity.RuntimeManager.CoreSystem.getMasterChannelGroup(out mcg);
        result = FMODUnity.RuntimeManager.CoreSystem.playSound(sound, mcg, false, out channel);

        // get master channel group fader
        FMOD.DSP mcgFader;
        result = mcg.getDSP(FMOD.CHANNELCONTROL_DSP_INDEX.FADER, out mcgFader);
        // get sound's fader
        FMOD.DSP channelFader;
        result = channel.getDSP(FMOD.CHANNELCONTROL_DSP_INDEX.HEAD, out channelFader);

        // create FFT DSP for left channel
        result = FMODUnity.RuntimeManager.CoreSystem.createDSPByType(FMOD.DSP_TYPE.FFT, out var lFFT);
        // add connection between sound channel fader and left FFT DSP
        result = lFFT.addInput(channelFader, out FMOD.DSPConnection lConnection, FMOD.DSPCONNECTION_TYPE.STANDARD);
        // set mix matrix on connection to remove right channel
        result = lConnection.setMixMatrix(new float[] { 1, 0 }, 1, 2);
        // set wet output of FFT DSP to 0, and set it to be active
        result = lFFT.setWetDryMix(1f, 0f, 0f);
        result = lFFT.setActive(true);
        // add out from DSP to master channel group fader
        result = mcgFader.addInput(lFFT, out _, FMOD.DSPCONNECTION_TYPE.STANDARD);

        // repeat the same for the right side, but swap the mix matrix so DSP only receives right channel
        result = FMODUnity.RuntimeManager.CoreSystem.createDSPByType(FMOD.DSP_TYPE.FFT, out var rFFT);
        result = rFFT.addInput(channelFader, out FMOD.DSPConnection rConnection, FMOD.DSPCONNECTION_TYPE.STANDARD);
        result = rConnection.setMixMatrix(new float[] { 0, 1 }, 1, 2);
        result = rFFT.setWetDryMix(1f, 0f, 0f);
        result = rFFT.setActive(true);
        result = mcgFader.addInput(rFFT, out _, FMOD.DSPCONNECTION_TYPE.STANDARD);
        sound.release();
        lFFT.release();
    }
    void OnDestroy()
    {
        
    }
}

Leah_FMOD · July 12, 2023, 11:54pm

The first warning indicates that your device is already in use, and as a result trying to stop it is warning you that the device hasn’t been initialized. it’s possible that the selected device is in exclusive mode, or that the wrong device has been selected.

That said, the code has issues, and fixing them may resolve the error/warning. sound and lFFT are being released in Start(), meaning that the code basically never gets to actually splitting the channels because the sound/FFT DSPs are likely destroyed before anything is played into them. rFFT is also never released. As mentioned previously, instead of declaring sound, rFFT, and lFFTinline inStart()`, they should be declared as class members, i.e.

public class thing1 : MonoBehaviour
{
    [SerializeField] int deviceIndex = 4;
    FMOD.DSP rFFT;
    FMOD.DSP lFFT;
    FMOD.Sound sound;

    //...
}

And they should be released in OnDestroy():

    void OnDestroy()
    {
        sound.release();
        lFFT.release();
        rFFT.release();
    }

If I fix those issues up and select the correct device, the code functions as expected.