I’ve been working on updating a simple voice chat script from using Unity’s built-in audio to using FMOD for features like spatialization and custom effects through a programmer instrument.
The main issue right now is turning these byte array chunks back into sound as they’re recieved.
What I’ve Tried
I’ve been successful in recording the player’s voice and playing it back as a singular recording Sound through a looping async programmer instrument, however this doesn’t work for my case because the audio is received in byte array segments over the network. This helped me learn a lot about the programmer instrument and callback.
I’ve tried modeling after the User Generated Sound example and made some progress, but the PCM callback functionality seems confusing when translating to C#, but it looks VERY similar to how I play a sound buffer through Unity’s AudioClip PCMReaderCallback. My original script converted the bytes back into a float array then wrote the data directly into an audio clip which is basically what I want to do to the programmer instrument’s sound.
Currently my updated code works with no crashes, however no audio is being played back. I assume because I need to be manually appending the bytes to a buffer instead of generating a new sound with the current buffer every time new data is received. I’ve implemented the PCM read and seek callbacks but I’m still not sure how to properly use them in this case. For instance, how do I work with the IntPtr arguments in the PCMRead and append the sound data as it’s received?
I’ve linked my original and modified (Fmod) scripts at the bottom.
The main two methods of interest are WriteToClip(byte uncompressed, int iSize), which is called when new data needs to be added to the sound buffer, and OnAudioRead(float data) which is the PCM reader callback for the original script which I assume gets replaced by fmod’s pcm read callback.
The updated script works by initializing an Fmod event and releasing manually when the player is destroyed, so that I can modify the event parameters during runtime. I’m modifying the recordedSound instance in WriteToClip and then reassigning it in the programmer callback. (I’m not sure how else to handle that.) It’s evident that I need to play these sounds from a buffer.
Any help or insight on how to get this working would be appreciated!
I’ve been researching this topic for a little more than a week now and would like to finally bridge the gap. Hopefully others could learn from it as well.