Mic state when recording

Unity
,
1

Hey there,

I recently posted a question and kept digging further. I now got a mic working with FMOD in Unity, and am able to hear myself. There is some weird stuff happening still though.
When I talk thru the microphone, it sounds very distorted. Another odd thing that happens is the recording looping back. So after a little while, I hear the recording of before basically.

Feel like I’m getting closer to my endpoint, but currently stuck on the mic sounding distorted and the looping.

A complete different question, is there a way to tell if a microphone is picking up sounds? Like is there a possibility to just add a boolean that reacts to input to the microphone? I know it’s a possibility with just playing back sounds, I did it with the getPlaybackState in another script that reads data from an audio clip.

Hope to hear from you guys,
Niels :slight_smile:

using UnityEngine;
using System;
using System.Text;  
using System.Runtime.InteropServices;

public class MicSound : MonoBehaviour
{
public bool m_isrecording;
public FMOD.Sound m_Sound;
public string mic;
public int systemRate, speakerModeChannels;
public FMOD.SPEAKERMODE speakerMode = FMOD.SPEAKERMODE.RAW;
public FMOD.DRIVER_STATE driverState = FMOD.DRIVER_STATE.DEFAULT;

public int numDriv, numCon;

FMOD.RESULT m_Result;
FMOD.MODE soundMode = FMOD.MODE.OPENUSER | FMOD.MODE.LOOP_NORMAL;
FMOD.CREATESOUNDEXINFO exinfo;
public byte[] m_Bytes;
public FMOD.ChannelGroup channelGroup;
public FMOD.Channel channel;

void Start()
{
    FMODUnity.RuntimeManager.CoreSystem.getRecordDriverInfo(0, out mic, 1, out Guid guid, out systemRate, out speakerMode, out speakerModeChannels, out driverState);
    FMODUnity.RuntimeManager.CoreSystem.getRecordNumDrivers(out numDriv, out numCon);

    exinfo.cbsize = Marshal.SizeOf(exinfo);
    exinfo.numchannels = speakerModeChannels;
    exinfo.format = FMOD.SOUND_FORMAT.PCM16;
    exinfo.defaultfrequency = systemRate;
    exinfo.length = (uint)(exinfo.defaultfrequency * exinfo.numchannels * sizeof(short)) * 8;

    byte[] m_Bytes = new byte[exinfo.length + 1];

    m_Result = FMODUnity.RuntimeManager.CoreSystem.createSound(m_Bytes, soundMode, ref exinfo, out m_Sound);

    if (m_Result != FMOD.RESULT.OK)
    {
        UnityEngine.Debug.Log("failed to createSound: " + m_Result);
        return;
    }

    m_Result = FMODUnity.RuntimeManager.CoreSystem.recordStart(0, m_Sound, false);

    if (m_Result != FMOD.RESULT.OK)
    {
        UnityEngine.Debug.Log("failed to startrecord: " + m_Result);
    }
    else
    {
        m_isrecording = true;
        UnityEngine.Debug.Log("Recording " + m_Result);
    }

    //test play
    FMODUnity.RuntimeManager.CoreSystem.playSound(m_Sound, channelGroup, false, out channel);
}
1 Like
2

This might help someone who has more knowledge about all the FMOD stuff, haha.

3

This sounds like it is because there is no latency between recording and playback.

https://fmod.com/resources/documentation-api?version=2.1&page=core-guide.html#recording-record-to-a-sound-from-microphone-or-line-in

The sound can also be played while it is recording, to allow realtime effects. A simple technique to achieve this is to start recording, then wait a small amount of time, like 50ms, then play the sound. This will keep the play cursor just behind the record cursor. See the record example for source and information on how to do this.

The record example can be found in the API download.

4

Hey,

Thans for your reply! It really helps.
Can you maybe paste the link for the API download? I can’t seem to find it on the Download page. Went thru the entire Unity Feature Demo, and there’s nothing like a record example in there.

Thanks again,
Niels :slight_smile:

5

Sorry it’s called the “FMOD Engine”, under “FMOD Studio Suite”.

6

Hey,

Thanks again for the help. Found the download with the examples you were talking about. Not something that will solve my problem, but that’s alright.
I added a delay on the moment the sound starts playing, and it now works fluently. There is only one thing left to do for me, and that’s to either get data out of my sound, or get like a status if sound is being made (like talking into the mic, or not).
I looked at the documentation, and tried to make my script work with readData, lock, unlock on the sound, but didn’t get the result I expected. Also tried to use the DSP_FFT.SPECTRUMDATA function, which I’m familiar with due to reading from a sound file, and making objects scale based on it’s volume. Also couldn’t get it to work properly.

Do you have an idea of what function to use, or is there an other simple solution for this?

Thanks again, already helped me a ton :slight_smile:
Niels

I know it won’t work with all the functions active, but here is the lines I used to test it out, for testing purposes, I activated them, but manually commented them out to actually test them.

void Update()
{
m_Sound.readData(buffer, length, out read);
m_Sound.@lock(offset, length, out ptr1, out ptr2, out len1, out len2);
m_Sound.unlock(ptr1, ptr2, len1, len2);
if (channelGroup.hasHandle())
{
channelGroup.isPlaying(out m_isPlaying);
}
if (m_isPlaying && m_dspAdded)
{
m_Result = m_DataDsp.getParameterData((int)FMOD.DSP_FFT.SPECTRUMDATA, out unmanagedData, out length);
m_DataParam = (FMOD.DSP_PARAMETER_FFT)Marshal.PtrToStructure(unmanagedData, typeof(FMOD.DSP_PARAMETER_FFT));
for (int bin = 1; bin < WINDOWSIZE; bin++)
{
float temp = lin2DB(m_DataParam.spectrum[0][bin]);
temp = ((temp + 80.0f) * (1 / 80.0f));
m_DataArray[bin] = Mathf.Lerp(m_DataArray[bin], temp, 0.6f);
}
}
}

7

There isn’t an api to find out the status of the sound being recorded, but using the spectrum data should work. What about it isn’t working? The code looks fine from what you have shown.

8

So, which method should I use? The spectrumdata one?
I’ve been trying to make it work properly, but can’t wrap my hands around it just yet.

if (m_dspAdded)
{
m_Result = m_DataDsp.getParameterData((int)FMOD.DSP_FFT.SPECTRUMDATA, out unmanagedData, out length);
Debug.Log("unmangData " + unmanagedData + " length " + length); //passes
m_DataParam = (FMOD.DSP_PARAMETER_FFT)Marshal.PtrToStructure(unmanagedData, typeof(FMOD.DSP_PARAMETER_FFT));
if (m_DataParam.numchannels < 1)
{
Debug.Log(“Failure”);
return;
}
Debug.Log("DataParam " + m_DataParam.spectrum[1][1]); //error
for (int bin = 1; bin < WINDOWSIZE; bin++)
{
float temp = lin2DB(m_DataParam.spectrum[0][bin]);
temp = ((temp + 80.0f) * (1 / 80.0f));
m_DataArray[bin] = Mathf.Lerp(m_DataArray[bin], temp, 0.6f);
}
}

Tried this, just to see where my problem occurs, and I guess it’s in the numChannels.
Since I’m asking for the script to debug, this is what it looks like as of right now:

Looks like it passes the getParameterData function, but gets stuck at the PtrToStructure function.
If I remove the numChannels check, it gives me the ‘Out of bounds error’, but that’s probably caused by the parameter not being set up properly. Do you have any idea on how to move on? Feel like I’m almost there!

Thanks :slight_smile:

9

Instead of doing this, you should check if there are more than zero channels before proceeding with the spectrum code. There may be frames where there is no data when the sound is starting up or stopping.

10

Ok, so I’ve been trying to work this out. No full luck yet, but I’m a little further than before.

I’ve compared the script that analyses the event sound and puts the data in an array, and I figured out how to properly pass thru the unmanaged data with the mic. The thing now is that I keep on getting an “Out of Bounds error” when it tries to parse it over to an readable array.

using UnityEngine;
using System;
using System.Text;
using System.Runtime.InteropServices;
using System.Collections;
public class MicSound : MonoBehaviour
{
// mic decs
public bool m_isrecording;
public FMOD.Sound m_Sound;
public string mic;
public int systemRate, speakerModeChannels;
public FMOD.SPEAKERMODE speakerMode = FMOD.SPEAKERMODE.RAW;
public FMOD.DRIVER_STATE driverState = FMOD.DRIVER_STATE.DEFAULT;
public int numDriv, numCon;
FMOD.RESULT m_Result;
FMOD.MODE soundMode = FMOD.MODE.OPENUSER | FMOD.MODE.LOOP_NORMAL;
FMOD.CREATESOUNDEXINFO exinfo;
// data decs
public float m_fftArray;
[FMODUnity.EventRef] public string m_eventRef;
FMOD.Studio.EventInstance m_eventInstance;
public FMOD.ChannelGroup m_channelGroup;
public FMOD.Channel channel;
FMOD.DSP m_fftDsp;
public bool m_dspAdded;
// general
public bool playSound;
public float delayTime = .05f;
public bool makingSound;
public int WINDOWSIZE = 1024;

void Awake()
{
    // * instance event
    m_eventInstance = FMODUnity.RuntimeManager.CreateInstance(m_eventRef);

    // set up mic
    FMODUnity.RuntimeManager.CoreSystem.getRecordDriverInfo(0, out mic, 1, out Guid guid, out systemRate, out speakerMode, out speakerModeChannels, out driverState);
    FMODUnity.RuntimeManager.CoreSystem.getRecordNumDrivers(out numDriv, out numCon);

    exinfo.cbsize = Marshal.SizeOf(exinfo);
    exinfo.numchannels = speakerModeChannels;
    exinfo.format = FMOD.SOUND_FORMAT.PCM16;
    exinfo.defaultfrequency = systemRate;
    exinfo.length = (uint)(exinfo.defaultfrequency * exinfo.numchannels * sizeof(short)) * 8;

    // * data array to be filled
    m_fftArray = new float[WINDOWSIZE];

    // making the mic sound
    m_Result = FMODUnity.RuntimeManager.CoreSystem.createSound(mic, soundMode, ref exinfo, out m_Sound);

    if (m_Result != FMOD.RESULT.OK)
    {
        UnityEngine.Debug.Log("failed to createSound: " + m_Result);
        return;
    }

    m_Result = FMODUnity.RuntimeManager.CoreSystem.recordStart(0, m_Sound, true);

    if (m_Result != FMOD.RESULT.OK)
    {
        UnityEngine.Debug.Log("failed to startrecord: " + m_Result);
    }
    else
    {
        m_isrecording = true;
        UnityEngine.Debug.Log("Recording " + m_Result);
    }

    // starting the event instance
    StartEventInstance();

    // adding a delay
    StartCoroutine(AddDelay(delayTime));
}

void Update()
{
    // dsp
    if (m_dspAdded && speakerModeChannels > 0)
    {
        IntPtr unmanagedData;
        uint length;

        m_Result = m_fftDsp.getParameterData((int)FMOD.DSP_FFT.SPECTRUMDATA, out unmanagedData, out length);

        Debug.Log("unmanData " + unmanagedData + " length " + length + " channels " + speakerModeChannels); //passes

        FMOD.DSP_PARAMETER_FFT m_DataParam = (FMOD.DSP_PARAMETER_FFT)Marshal.PtrToStructure(unmanagedData, typeof(FMOD.DSP_PARAMETER_FFT));

        if (m_DataParam.numchannels > 0)
        {
            Debug.Log("Failure");
            return;
        }

        // Debug.Log("DataParam " + m_DataParam.spectrum[0][0]); //error

        for (int bin = 0; bin < WINDOWSIZE; bin++)
        {
            float temp = lin2DB(m_DataParam.spectrum[0][bin]);
            temp = ((temp + 80.0f) * (1 / 80.0f));
            m_fftArray[bin] = Mathf.Lerp(m_fftArray[bin], temp, 0.6f);
        }
    }
}

bool StartEventInstance()
{
    m_eventInstance.start();
    return true;
}

IEnumerator AddDelay(float seconds)
{
    yield return new WaitForSeconds(seconds);
    if (playSound)
    {
        FMODUnity.RuntimeManager.CoreSystem.playSound(m_Sound, m_channelGroup, false, out channel);
    }

    // dsp
    StartCoroutine(AddDspToChannel());
    m_eventInstance = FMODUnity.RuntimeManager.CreateInstance(m_eventRef);
}

float lin2DB(float linear)
{
    return (Mathf.Clamp(Mathf.Log10(linear) * 20, -80.0f, 0.0f));
}

IEnumerator AddDspToChannel()
{
    while (!m_channelGroup.hasHandle())
    {
        m_eventInstance.getChannelGroup(out m_channelGroup);
        yield return null;
    }

    FMODUnity.RuntimeManager.CoreSystem.createDSPByType(FMOD.DSP_TYPE.FFT, out m_fftDsp);
    m_fftDsp.setParameterInt((int)FMOD.DSP_FFT.WINDOWTYPE, (int)FMOD.DSP_FFT_WINDOW.RECT);
    m_fftDsp.setParameterInt((int)FMOD.DSP_FFT.WINDOWSIZE, WINDOWSIZE * 2);

    m_channelGroup.addDSP(FMOD.CHANNELCONTROL_DSP_INDEX.TAIL, m_fftDsp);
    m_dspAdded = true;
}

void OnDestroy()
{
    m_eventInstance.stop(FMOD.Studio.STOP_MODE.IMMEDIATE);
    m_eventInstance.release();
}}

Here’s a video where I notice the difference. In the script you’ve made, the parsing works perfectly fine, but when I try to do the same, with the (as far as I can tell) same declarations and code used, it just doesnt parse. Debugging the raw data also works on the code you’ve written, but not on mine.

Hope to hear from you soon,
Niels :slight_smile:

11

Hey, at it again.
Been trying to debug the channels now, since my solution above isn’t correct I assume.

int test;
m_Result = m_channelGroup.getNumChannels(out test);
Debug.Log("numchannels " + test);
m_Result = FMODUnity.RuntimeManager.CoreSystem.getChannelsPlaying(out test);
Debug.Log("channelsplaying " + test);

Putting this code in Update results in this:

Does that help?

12

Hello,
I was wondering whether you ever got this fully working in Unity, and if you’d be willing to share the final implementation? The information I can find on implementing the Mic is scattered and fragmented, and this thread seems to be the best information that I could find.

1 Like