Trouble playing data segments through Programmer Instrument

Unity
1

Main Issue
I’ve been working on updating a simple voice chat script from using Unity’s built-in audio to using FMOD for features like spatialization and custom effects through a programmer instrument.
The main issue right now is turning these byte array chunks back into sound as they’re recieved.

What I’ve Tried
I’ve been successful in recording the player’s voice and playing it back as a singular recording Sound through a looping async programmer instrument, however this doesn’t work for my case because the audio is received in byte array segments over the network. This helped me learn a lot about the programmer instrument and callback.

I’ve tried modeling after the User Generated Sound example and made some progress, but the PCM callback functionality seems confusing when translating to C#, but it looks VERY similar to how I play a sound buffer through Unity’s AudioClip PCMReaderCallback. My original script converted the bytes back into a float array then wrote the data directly into an audio clip which is basically what I want to do to the programmer instrument’s sound.

Currently my updated code works with no crashes, however no audio is being played back. I assume because I need to be manually appending the bytes to a buffer instead of generating a new sound with the current buffer every time new data is received. I’ve implemented the PCM read and seek callbacks but I’m still not sure how to properly use them in this case. For instance, how do I work with the IntPtr arguments in the PCMRead and append the sound data as it’s received?

I’ve linked my original and modified (Fmod) scripts at the bottom.
The main two methods of interest are WriteToClip(byte[] uncompressed, int iSize), which is called when new data needs to be added to the sound buffer, and OnAudioRead(float[] data) which is the PCM reader callback for the original script which I assume gets replaced by fmod’s pcm read callback.

The updated script works by initializing an Fmod event and releasing manually when the player is destroyed, so that I can modify the event parameters during runtime. I’m modifying the recordedSound instance in WriteToClip and then reassigning it in the programmer callback. (I’m not sure how else to handle that.) It’s evident that I need to play these sounds from a buffer.

Any help or insight on how to get this working would be appreciated!
I’ve been researching this topic for a little more than a week now and would like to finally bridge the gap. Hopefully others could learn from it as well.

Code Links
Original Script (Built-in audio): https://pastebin.com/WMvVWMu1
Modified Script (Using Fmod): https://pastebin.com/9zuqtD9g

2

That all seems reasonable, the thing to do now to get audio playing back is to feed the pcmread callback’s data with the data you are getting from the server. e.g

[AOT.MonoPInvokeCallback(typeof(FMOD.SOUND_PCMREAD_CALLBACK))]
static FMOD.RESULT PcmReadCallback(IntPtr sound, IntPtr data, uint datalen)
{
    int arraySize = (int)(datalen >> 2); // Convert length in byte to length in short

    short[] stereo16bitbuffer = new short[arraySize];

    for (int i = 0; i < arraySize; ++i)
    {
        if (buffer.Count > 0) // buffer is a float queue being fed audio data from the server
        {
            stereo16bitbuffer[i] = (short)(buffer.Dequeue() * 32767); // basic float to int16
        }
        else
        {
            // Buffer starvation, should alert the user
        }
    }

    Marshal.Copy(stereo16bitbuffer, 0, data, arraySize);

    return FMOD.RESULT.OK;
}

In this case I am just pushing the audio data from the source to a Queue as a quick and easy buffer to read from in the callback, but a Circular Buffer would be better.
Hopefully that gives you a basic idea of how to fill the PcmReadCallback in C#, let me know if you have any other questions!

3

Hi Jeff,
I managed to get it working a while ago through some experimentation and it seems to do the trick!
I wont paste the full code but this is the PCM reader callback I ended up with and I ended up only initializing the sound once during the programmer instrument creation callback.

    FMOD.RESULT PcmReadCallback(IntPtr sound, IntPtr data, uint datalen) {
        if(!isReady || outputBuffer == null || readerPosition>outputBuffer.Length) return FMOD.RESULT.OK;

        byte[] modifiedData = new byte[datalen];

        for (int i=0; i<datalen; ++i)
        {
            // Start with silence
            modifiedData[i] = 0;
 
            // Check for any buffer data
            if (bufferCount > 0)
            {
                // Copy data from reader position
                modifiedData[i] = outputBuffer[readerPosition];

                // Loop reader index
                readerPosition = (readerPosition + 1) % outputBufferSize;
                bufferCount--;
            }
        }

        Marshal.Copy(modifiedData, 0, data, (int)datalen);

        return FMOD.RESULT.OK;
    }
1 Like