# Using FMOD with voice chat utilizing OnAudioFilterRead() **URL:** \ **Category:** Unity\ **Created:** [October 8, 2021, 5:02am UTC](https://qa.fmod.com/t/using-fmod-with-voice-chat-utilizing-onaudiofilterread/17763 "2021-10-08T05:02:32Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![jessekirbs](https://avatars.discourse-cdn.com/v4/letter/j/6de8d8/32.png) [@jessekirbs](https://qa.fmod.com/u/jessekirbs)\ **Post date:** [October 8, 2021, 5:02am UTC](https://qa.fmod.com/t/using-fmod-with-voice-chat-utilizing-onaudiofilterread/17763/1 "2021-10-08T05:02:32Z")
I’m trying to integrate Normcore multiplayer SDK into our project. They offer low-latency voice chat by utilizing OnAudioFilterRead(), playing a dummy clip, and then use an audio effect to inject voice data. I’d like to use the FMOD snapshots in our project with the voice chat system - is this possible given the above info? Or is OnAudioFilterRead() strictly for use with the Unity audio engine and therefore not accessible with FMOD? Any guidance would be greatly appreciated. Thank you! ---
**Author:** ![jeff\_fmod](https://yyz2.discourse-cdn.com/flex036/user_avatar/qa.fmod.com/jeff_fmod/32/1766_2.png) [@jeff\_fmod](https://qa.fmod.com/u/jeff_fmod)\ **Post date:** [October 12, 2021, 11:45pm UTC](https://qa.fmod.com/t/using-fmod-with-voice-chat-utilizing-onaudiofilterread/17763/2 "2021-10-12T23:45:37Z")
You should be able to grab data from OnAudioFilterRead and send it through FMOD using a programmer instrument, thus allowing you to take advantage of snapshots. The process would be something like this: - Setup a [programmer instrument](https://www.fmod.com/resources/documentation-studio?version=2.02&page=glossary.html#programmer-instrument) in FMOD studio - Setup your effects, snapshots etc - Grab your mic data from `OnAudioFilterRead` - Load the data into an FMOD.Sound, either using [`Sound.lock and Sound.unlock`](https://fmod.com/resources/documentation-api?version=2.02&page=core-api-sound.html#sound_lock) or implementing a [`FMOD_SOUND_PCM_READ_CALLBACK`](https://www.fmod.com/resources/documentation-api?version=2.02&page=core-api-sound.html#fmod_sound_pcmread_callback) - Play that sound through a programmer instrument that you setup in FMOD studio - [Play the programmer](https://www.fmod.com/resources/documentation-unity?version=2.02&page=examples-programmer-sounds.html) sound in Unity - Call your snapshots, param changes etc ---
**Author:** ![jessekirbs](https://avatars.discourse-cdn.com/v4/letter/j/6de8d8/32.png) [@jessekirbs](https://qa.fmod.com/u/jessekirbs)\ **Post date:** [October 14, 2021, 10:50pm UTC](https://qa.fmod.com/t/using-fmod-with-voice-chat-utilizing-onaudiofilterread/17763/3 "2021-10-14T22:50:11Z")
@jeff_fmod Thank you very much for the reply! We’ll try this out and report back. ---
**Author:** ![kantagara](https://yyz2.discourse-cdn.com/flex036/user_avatar/qa.fmod.com/kantagara/32/1666_2.png) [@kantagara](https://qa.fmod.com/u/kantagara)\ **Post date:** [October 28, 2021, 8:27pm UTC](https://qa.fmod.com/t/using-fmod-with-voice-chat-utilizing-onaudiofilterread/17763/4 "2021-10-28T20:27:10Z")
Hey Jeff, thanks for this, I was looking into this because I’m working on something similar but I’ve now got into the wall because these examples weren’t that clear to me tbh ```auto { uint lenbytes = (uint)(data.Length * channels * sizeof(float)); Sound sound = new Sound(); var result = sound.@lock(0, lenbytes, out var ptr1, out var ptr2, out var len1, out var len2); if (result != RESULT.OK) { Debug.LogError("Error! " + result); return; } var instance = FMODUnity.RuntimeManager.CreateInstance("event:/Music_Mixed/User_Voice"); //What to actually pass in here, I have two potential IntPtrs and setUserData only accepts one instance.setUserData() ``` Any help would be much appreciated. Thanks! ---
**Author:** ![jeff\_fmod](https://yyz2.discourse-cdn.com/flex036/user_avatar/qa.fmod.com/jeff_fmod/32/1766_2.png) [@jeff\_fmod](https://qa.fmod.com/u/jeff_fmod)\ **Post date:** [October 31, 2021, 11:49pm UTC](https://qa.fmod.com/t/using-fmod-with-voice-chat-utilizing-onaudiofilterread/17763/5 "2021-10-31T23:49:17Z")
If you find yourself needing to pass two values into `setUserData`, you can just create a new class that holds both values- this is what we do in the [Timeline Example](https://fmod.com/resources/documentation-unity?version=2.02&page=examples-timeline-callbacks.html) by creating the `Timelineinfo` class to hold both the bar and marker values. ---
**Author:** ![kantagara](https://yyz2.discourse-cdn.com/flex036/user_avatar/qa.fmod.com/kantagara/32/1666_2.png) [@kantagara](https://qa.fmod.com/u/kantagara)\ **Post date:** [November 2, 2021, 5:48pm UTC](https://qa.fmod.com/t/using-fmod-with-voice-chat-utilizing-onaudiofilterread/17763/6 "2021-11-02T17:48:09Z")
Hey Jeff, thanks for the answer! Figued out that part but am having actually a hard time creating the sound object ```auto void OnAudioFilterRead(float[] data, int channels) { if(data == null || data.Length == 0) return; var result = ConvertArrayToFMODSound(data, channels, out var sound); if (result != RESULT.OK) { Debug.LogError($"Error creating the sound! Got this from the FMOD {result}"); return; } //Optimize this with array of event instances pool? var programmerSound = RuntimeManager.CreateInstance("event:/Music_Mixed/User_Voice"); GCHandle soundHandle = GCHandle.Alloc(sound, GCHandleType.Pinned); programmerSound.setUserData(GCHandle.ToIntPtr(soundHandle)); programmerSound.setCallback(audioCallback); programmerSound.start(); programmerSound.release(); } private RESULT ConvertArrayToFMODSound(float[] data, int channels, out Sound sound) { //Tried with data.length * channels as well, but same results uint lenBytes = (uint)(data.Length * sizeof(float)); CREATESOUNDEXINFO soundInfo = new CREATESOUNDEXINFO(); soundInfo.length = lenBytes; soundInfo.format = SOUND_FORMAT.PCMFLOAT; soundInfo.numchannels = channels; //What should be set in here? //soundInfo.defaultfrequency = ?? RuntimeManager.CoreSystem.createSound("", MODE.OPENUSER, ref soundInfo, out sound); IntPtr ptr1, ptr2; uint len1, len2; sound.@lock(0, lenBytes, out ptr1, out ptr2, out len1, out len2); Marshal.Copy(data, 0, ptr1, (int)(len1 / sizeof(float))); if (len2 > 0) { Marshal.Copy(data, (int)(len1 / sizeof(float)), ptr2, (int)(len2 / sizeof(float))); } sound.unlock(ptr1, ptr2, len1, len2); Debug.Log(sound.hasHandle()); var result = sound.setMode(MODE.LOOP_NORMAL); return result; } [MonoPInvokeCallback(typeof(EVENT_CALLBACK))] static RESULT AudioEventCallback(EVENT_CALLBACK_TYPE type, IntPtr instancePtr, IntPtr parameterPtr) { EventInstance instance = new EventInstance(instancePtr); // Retrieve the user data instance.getUserData(out var pointer); // Get the string object GCHandle stringHandle = GCHandle.FromIntPtr(pointer); Sound sound = (Sound)stringHandle.Target; switch (type) { case EVENT_CALLBACK_TYPE.CREATE_PROGRAMMER_SOUND: { var parameter = (PROGRAMMER_SOUND_PROPERTIES)Marshal.PtrToStructure(parameterPtr, typeof(PROGRAMMER_SOUND_PROPERTIES)); parameter.sound = sound.handle; parameter.subsoundIndex = -1; Marshal.StructureToPtr(parameter, parameterPtr, false); break; } case EVENT_CALLBACK_TYPE.DESTROY_PROGRAMMER_SOUND: { var parameter = (PROGRAMMER_SOUND_PROPERTIES)Marshal.PtrToStructure(parameterPtr, typeof(PROGRAMMER_SOUND_PROPERTIES)); sound.release(); sound = new Sound(parameter.sound); sound.release(); break; } case EVENT_CALLBACK_TYPE.DESTROYED: { // Now the event has been destroyed, unpin the string memory so it can be garbage collected stringHandle.Free(); break; } } return RESULT.OK; } } ``` I always get the INVALID\_PARAM Error from the RESULT on the ConvertArrayToFMODSound. Any help would me much appreciated! ---
**Author:** ![jeff\_fmod](https://yyz2.discourse-cdn.com/flex036/user_avatar/qa.fmod.com/jeff_fmod/32/1766_2.png) [@jeff\_fmod](https://qa.fmod.com/u/jeff_fmod)\ **Post date:** [November 3, 2021, 12:50am UTC](https://qa.fmod.com/t/using-fmod-with-voice-chat-utilizing-onaudiofilterread/17763/7 "2021-11-03T00:50:24Z")
> [@kantagara](#): > > INVALID\_PARAM Error from the RESULT on the ConvertArrayToFMODSound I think this is because you are missing the cbsize field which just needs to be set to the size of `CREATESOUNDEXINFO` object. e.g ```auto soundInfo.cbsize = Marshal.SizeOf(typeof(FMOD.CREATESOUNDEXINFO)); ``` > [@kantagara](#): > > ```auto > //What should be set in here? > //soundInfo.defaultfrequency = ?? > > ``` This would usually be either 44100 to 48000, in any case it ought to match the sample rate your audio card is using. In Unity this is stored in `AudioSettings.outputSampleRate`. > [@kantagara](#): > > ```auto > //Optimize this with array of event instances pool? > > ``` I agree perhaps some instance pools or storing the audio from `OnAudioFilterRead` in a ring buffer and periodically firing off instances, but optimizations are for after it’s working! ---
**Author:** ![kantagara](https://yyz2.discourse-cdn.com/flex036/user_avatar/qa.fmod.com/kantagara/32/1666_2.png) [@kantagara](https://qa.fmod.com/u/kantagara)\ **Post date:** [November 3, 2021, 8:14am UTC](https://qa.fmod.com/t/using-fmod-with-voice-chat-utilizing-onaudiofilterread/17763/8 "2021-11-03T08:14:23Z")
Thank you so much for the answer, we’re getting there! 😄 ```auto ArgumentException: start_index + length > array length ``` ```auto //Changed it from data.Length to systemSampleRate, still getting that exception uint lenBytes = (uint)(_systemSampleRate* channels * sizeof(float)); CREATESOUNDEXINFO soundInfo = new CREATESOUNDEXINFO(); soundInfo.length = lenBytes; soundInfo.format = SOUND_FORMAT.PCMFLOAT; soundInfo.numchannels = channels; soundInfo.defaultfrequency = _systemSampleRate; soundInfo.cbsize = Marshal.SizeOf(typeof(FMOD.CREATESOUNDEXINFO)); RuntimeManager.CoreSystem.createSound("", MODE.OPENUSER, ref soundInfo, out sound); IntPtr ptr1, ptr2; uint len1, len2; sound.@lock(0, lenBytes, out ptr1, out ptr2, out len1, out len2); //Exception happens below Marshal.Copy(data, 0, ptr1, (int)(len1 / sizeof(float))); if (len2 > 0) { Marshal.Copy(data, (int)(len1 / sizeof(float)), ptr2, (int)(len2 / sizeof(float))); } ``` I have no idea what I’m doing wrong, if you could point it out it would be amazing. Thanks again! ---
**Author:** ![jeff\_fmod](https://yyz2.discourse-cdn.com/flex036/user_avatar/qa.fmod.com/jeff_fmod/32/1766_2.png) [@jeff\_fmod](https://qa.fmod.com/u/jeff_fmod)\ **Post date:** [November 4, 2021, 3:39am UTC](https://qa.fmod.com/t/using-fmod-with-voice-chat-utilizing-onaudiofilterread/17763/9 "2021-11-04T03:39:14Z")
> [@kantagara](#): > > ```auto > //Changed it from data.Length to systemSampleRate, still getting that exception > uint lenBytes = (uint)(_systemSampleRate* channels * sizeof(float)); > > ``` You had it correctly before, should be ```auto uint lenBytes = (uint)(data.Length * sizeof(float)); ``` Other things to check: - Make sure \_systemSampleRate has a non-zero value - Put an error check on the `CoreSystem.createSound` line and return early if it failed ---
**Author:** ![kantagara](https://yyz2.discourse-cdn.com/flex036/user_avatar/qa.fmod.com/kantagara/32/1666_2.png) [@kantagara](https://qa.fmod.com/u/kantagara)\ **Post date:** [November 7, 2021, 9:55pm UTC](https://qa.fmod.com/t/using-fmod-with-voice-chat-utilizing-onaudiofilterread/17763/10 "2021-11-07T21:55:27Z")
Well combining Normcores functions with this did in fact made the sound go trough FMOD! So thank you so much for that! So there is another issue which I think is a bit trickier here… spatial audio is completely gone when RealTimeAvatarVoice component is attached, that component is also responsible for the audio chat itself… I don’t see any reason why that would be mutually exclusive so I’m sending both of the scripts now, maybe you could see some clear indications what’s brekaing. When the script is not there, spatial audio works, but voice chat isn’t… [Greenlight 1 - Pastebin.com](https://pastebin.com/E7JRAM1C) This is audio output script [Greenlight 2 - Pastebin.com](https://pastebin.com/VPNyPw4S) ---
**Author:** ![jessekirbs](https://avatars.discourse-cdn.com/v4/letter/j/6de8d8/32.png) [@jessekirbs](https://qa.fmod.com/u/jessekirbs)\ **Post date:** [November 7, 2021, 11:46pm UTC](https://qa.fmod.com/t/using-fmod-with-voice-chat-utilizing-onaudiofilterread/17763/11 "2021-11-07T23:46:58Z")
Hey @jeff_fmod - Just to clarify what @kantagara is saying: he got the scripts working to the point where it seems like we’re getting input and output from the mics across app instances, but this is only working with ‘Disable Unity Audio’ UNchecked. When this is unchecked, it breaks all the spatial FMOD events and snapshots, and everything is 2D. But when it’s checked (as FMOD asks you to do in the setup process), the voicechat scripts do not work. We’re confused about how to integrate the voicechat without breaking the FMOD spatial/snapshot setup. Thanks for your continued help. ---
**Author:** ![jeff\_fmod](https://yyz2.discourse-cdn.com/flex036/user_avatar/qa.fmod.com/jeff_fmod/32/1766_2.png) [@jeff\_fmod](https://qa.fmod.com/u/jeff_fmod)\ **Post date:** [November 8, 2021, 3:07am UTC](https://qa.fmod.com/t/using-fmod-with-voice-chat-utilizing-onaudiofilterread/17763/12 "2021-11-08T03:07:37Z")
Leaving Unity Audio enabled would really only be a problem on Xbox, it looks like you are using Android/Oculus so it shouldn’t cause any issues leaving it enabled and it certainly shouldn’t be breaking FMOD’s spatialization. When you say it’s breaking all spatilization, do you mean even events played on other objects in different locations are coming out 2D as well or is it just the programmer event / mic audio that you can’t get spatialization on? In any case, I have not been able to reproduce the issue with spatialization on my end. To start with, in GreenLightAudioOutput on line 159: ```auto programmerSound.set3DAttributes(position.To3DAttributes()); ``` Can you please confirm that `position.To3DAttributes()` is returning the correct location of the gameObject and not just {0,0,0} or something like that? ---
**Author:** ![jessekirbs](https://avatars.discourse-cdn.com/v4/letter/j/6de8d8/32.png) [@jessekirbs](https://qa.fmod.com/u/jessekirbs)\ **Post date:** [November 8, 2021, 6:40am UTC](https://qa.fmod.com/t/using-fmod-with-voice-chat-utilizing-onaudiofilterread/17763/13 "2021-11-08T06:40:34Z")
@jeff_fmod I’m so sorry - I jumped in and made this more confusing. I’m wrong - it’s not the ‘Disable Unity Audio’ checkbox that’s breaking the spatialization. It’s only the script that @kantagara posted. We are using iOS. ---
**Author:** ![kantagara](https://yyz2.discourse-cdn.com/flex036/user_avatar/qa.fmod.com/kantagara/32/1666_2.png) [@kantagara](https://qa.fmod.com/u/kantagara)\ **Post date:** [November 8, 2021, 8:43pm UTC](https://qa.fmod.com/t/using-fmod-with-voice-chat-utilizing-onaudiofilterread/17763/14 "2021-11-08T20:43:23Z")
@jeff_fmod actually the thing that breaks spatialization is the GreenLightRealTimeAvatarVoice, when we remove that component, spatialization works as expected, but that thing is needed in order for realtime voice chat to work properly. To answer your question, set3d attribute uses appropriate values and it’s not vector3.zero because I’m caching position in update loop. Did debug logged it and it confirmed it as well ---
**Author:** ![jeff\_fmod](https://yyz2.discourse-cdn.com/flex036/user_avatar/qa.fmod.com/jeff_fmod/32/1766_2.png) [@jeff\_fmod](https://qa.fmod.com/u/jeff_fmod)\ **Post date:** [November 9, 2021, 1:48am UTC](https://qa.fmod.com/t/using-fmod-with-voice-chat-utilizing-onaudiofilterread/17763/15 "2021-11-09T01:48:39Z")
Thank you both for clarifying there- I believe I have reproduced the issue using the scripts you provided. Do you have listeners on your Player prefabs? If so, I have found that removing the listeners from new arrivals (guests?) has allowed spatialization to behave as expected. Perhaps you could make some kind of manager where you remove listeners from everything that isn’t your player? If that doesn’t apply to your setup / doesn’t fix your issue I might need a repro to get a better understanding of how your scene is setup. ---
**Author:** ![kantagara](https://yyz2.discourse-cdn.com/flex036/user_avatar/qa.fmod.com/kantagara/32/1666_2.png) [@kantagara](https://qa.fmod.com/u/kantagara)\ **Post date:** [November 9, 2021, 12:26pm UTC](https://qa.fmod.com/t/using-fmod-with-voice-chat-utilizing-onaudiofilterread/17763/16 "2021-11-09T12:26:18Z")
We don’t have more than 1 audio listener per the entire scene, it’s attached to one single object that’s not getting instantiated for every arriving client, and this in fact happens when there’s only just one player. Are you talking about the audio listener only, or some other type of listener? ---
**Author:** ![kantagara](https://yyz2.discourse-cdn.com/flex036/user_avatar/qa.fmod.com/kantagara/32/1666_2.png) [@kantagara](https://qa.fmod.com/u/kantagara)\ **Post date:** [November 12, 2021, 8:36am UTC](https://qa.fmod.com/t/using-fmod-with-voice-chat-utilizing-onaudiofilterread/17763/18 "2021-11-12T08:36:56Z")
@jeff_fmod any luck with reproing the issue? ---
**Author:** ![jeff\_fmod](https://yyz2.discourse-cdn.com/flex036/user_avatar/qa.fmod.com/jeff_fmod/32/1766_2.png) [@jeff\_fmod](https://qa.fmod.com/u/jeff_fmod)\ **Post date:** [November 15, 2021, 4:56am UTC](https://qa.fmod.com/t/using-fmod-with-voice-chat-utilizing-onaudiofilterread/17763/19 "2021-11-15T04:56:43Z")
I have reproduced the issue on iOS as you’ve described, I have found that it also reproduces with regular Unity Audio components (Audio Listener, Audio Source with 3D spatial blend) so this doesn’t seem to be an issue with FMOD. On my device spatial audio exists for a second or so, ceases when the VR Player arrives, and returns if I leave the app and re-enter, is this the case for you too? ---
**Author:** ![kantagara](https://yyz2.discourse-cdn.com/flex036/user_avatar/qa.fmod.com/kantagara/32/1666_2.png) [@kantagara](https://qa.fmod.com/u/kantagara)\ **Post date:** [November 18, 2021, 8:56am UTC](https://qa.fmod.com/t/using-fmod-with-voice-chat-utilizing-onaudiofilterread/17763/20 "2021-11-18T08:56:44Z")
Hey @jeff_fmod thanks a bunch for getting that repro. Indeed the case is the same for us, we’ll get in touch with normcore so they can fix the issue. And of course, everything you said for how to get the audio working properly is indeed working, with a minor modification where I had to divide the default frequency with the number of channels, then everything indeed started utilizing FMODs audio properly (we could hear the reverb and everything). Pasting the code here so some other fellow developer might find it useful 🙂 ```auto //audiodata is first processed via normcores audiooutputstream so we get the proper audio data back, that was also one of the issues why I didn't get the sound from the data[] array in onaudiofilterread var result = ConvertArrayToFMODSound(_audioData, channels, out var sound); if (result != RESULT.OK) { Debug.LogError($"Error creating the sound! Got this from the FMOD {result}"); return; } var programmerSound = RuntimeManager.CreateInstance("event:/Music_Mixed/User_Voice"); programmerSound.set3DAttributes(position.To3DAttributes()); GCHandle soundHandle = GCHandle.Alloc(sound, GCHandleType.Pinned); programmerSound.setUserData(GCHandle.ToIntPtr(soundHandle)); programmerSound.setCallback(audioCallback); programmerSound.start(); programmerSound.release(); private RESULT ConvertArrayToFMODSound(float[] data, int channels, out Sound sound) { uint lenBytes = (uint)(data.Length * sizeof(float)); CREATESOUNDEXINFO soundInfo = new CREATESOUNDEXINFO(); soundInfo.length = lenBytes; soundInfo.format = SOUND_FORMAT.PCMFLOAT; soundInfo.numchannels = channels; soundInfo.defaultfrequency = _systemSampleRate / channels; soundInfo.cbsize = Marshal.SizeOf(typeof(FMOD.CREATESOUNDEXINFO)); var res =RuntimeManager.CoreSystem.createSound("", MODE.OPENUSER, ref soundInfo, out sound); if (_systemSampleRate == 0) { Debug.LogError("Umm system sample rate is 0, that's odd"); return RESULT.ERR_FORMAT; } if (res != RESULT.OK) { Debug.LogError($"Result is not valid. It should be OK but it's {res}"); return res; } IntPtr ptr1, ptr2; uint len1, len2; sound.@lock(0, lenBytes, out ptr1, out ptr2, out len1, out len2); Marshal.Copy(data, 0, ptr1, (int)(len1 / sizeof(float))); if (len2 > 0) { Marshal.Copy(data, (int)(len1 / sizeof(float)), ptr2, (int)(len2 / sizeof(float))); } sound.unlock(ptr1, ptr2, len1, len2); var result = sound.setMode(MODE.LOOP_NORMAL); return result; } ``` ---
**Author:** ![jessekirbs](https://avatars.discourse-cdn.com/v4/letter/j/6de8d8/32.png) [@jessekirbs](https://qa.fmod.com/u/jessekirbs)\ **Post date:** [November 24, 2021, 12:19am UTC](https://qa.fmod.com/t/using-fmod-with-voice-chat-utilizing-onaudiofilterread/17763/22 "2021-11-24T00:19:59Z")
Hey @jeff_fmod - would you mind telling me how to reproduce your steps of testing this with Unity Audio? Thanks! [Next page](https://qa.fmod.com/t/using-fmod-with-voice-chat-utilizing-onaudiofilterread/17763.md?page=2)