Playback raw PCM bytes without buffering

Unity
1

I’m working on what will eventually be a tool to stream user audio input in realtime between servers. So my program is reading in mic data into one byte array (using @lock/unlock), and then another FMOD::Sound will playback audio from that array.
My understanding is that to read in raw PCM bytes, FMOD.MODE.CREATESTREAM is needed, which relies on buffering before playback. This is creating latency that I can not get rid of. I’ve tried waiting before calling playSound(), similar to the get_user_sound sample. Along with adjusting decodebuffersize. This helped a little, but not enough. And making it too small lead to stuttering.

How can play back the PCM bytes in real time without buffering delay? Without adjusting decodebuffersize, it’s ~1000ms delay. With adjustments to decodebuffersize, I got it down to ~300ms, but this is too slow.

Input mic code snippets:

void Start()
{
    // Store relevant information into FMOD.CREATESOUNDEXINFO variable.
    exinfo.cbsize = Marshal.SizeOf(typeof(FMOD.CREATESOUNDEXINFO));
    exinfo.numchannels = NumOfChannels;
    exinfo.format = FMOD.SOUND_FORMAT.PCM16;
    exinfo.defaultfrequency = SampleRate;
    exinfo.length = ((uint)SampleRate * sizeof(short) * (uint)NumOfChannels);
    var res = RuntimeManager.CoreSystem.createSound(exinfo.userdata, FMOD.MODE.LOOP_NORMAL | FMOD.MODE.OPENUSER,
            ref exinfo, out sound);
    FMOD_ERRCHECK(res);

    // Start recording
    FMOD_ERRCHECK(RuntimeManager.CoreSystem.recordStart(RecordingDeviceIndex, sound, true));
    FMOD_ERRCHECK(sound.getLength(out soundLength, FMOD.TIMEUNIT.PCM));
}

void Update()
{
    FMOD_ERRCHECK(RuntimeManager.CoreSystem.getRecordPosition(RecordingDeviceIndex, out recordpos));
    if (recordpos != lastrecordpos)
    {
        int blocklength;
        uint len1, len2;
        blocklength = (int)recordpos - (int)lastrecordpos;
        if (blocklength < 0)
        {
            blocklength += (int)soundLength;
        }
        FMOD_ERRCHECK(sound.@lock(lastrecordpos * (uint)exinfo.numchannels * sizeof(short), (uint)blocklength * (uint)exinfo.numchannels * sizeof(short), out ptr1, out ptr2, out len1, out len2));
        if (len1 > 0)
        {
            if (len1 + samplePos > soundData.Length)
            {
                if (len1 > samplePos)
                {
                    samplePos = 0;
                }
                else
                {
                    // Remove oldest sound first
                    shiftArrayStart(ref soundData, len1);
                    samplePos -= (int)len1;
                }
            }
            //Debug.Log("Copying " + len1 + " bytes to position " + samplePos);
            Marshal.Copy(ptr1, soundData, samplePos, (int)len1);
            samplePos += (int)len1;
        }
        if (len2 > 0)
        {
            // not handled...
        }
        FMOD_ERRCHECK(sound.unlock(ptr1, ptr2, len1, len2));
    }
    lastrecordpos = recordpos;
}

Playback sound code snippets:

void Start()
{
    //Setup receive stream to playback sound
    exinfo2.cbsize = Marshal.SizeOf(typeof(FMOD.CREATESOUNDEXINFO));
    exinfo2.numchannels = NumOfChannels;
    exinfo2.format = FMOD.SOUND_FORMAT.PCM16;
    exinfo2.defaultfrequency = SampleRate;
    exinfo2.pcmreadcallback = PCMREADCALLBACK;
    exinfo2.length = exinfo.length;
    exinfo2.decodebuffersize = 4096;
    FMOD_ERRCHECK(RuntimeManager.CoreSystem.createSound(exinfo2.userdata, FMOD.MODE.LOOP_NORMAL | FMOD.MODE.OPENUSER |FMOD.MODE.OPENONLY | FMOD.MODE.CREATESTREAM, ref exinfo2, out recvSound));
    FMOD_ERRCHECK(RuntimeManager.CoreSystem.playSound(recvSound, channelGroup, true, out channel));
}

private FMOD.RESULT PCMREADCALLBACK(IntPtr soundraw, IntPtr data, uint sizebytes)
{
    if (samplePos == 0)
    {
        // nothing recorded yet
        return FMOD.RESULT.OK;
    }
    if (sizebytes >= samplePos)
    {
        // Copy everything
        Marshal.Copy(soundData, 0, data, samplePos);
        samplePos = 0;
    }
    else
    {
        // Only copy what fits
        Marshal.Copy(soundData, 0, data, (int)sizebytes);
        // shift whatevers left to the start
        shiftArrayStart(ref soundData, sizebytes);
        samplePos -= (int)sizebytes;
    }
    return FMOD.RESULT.OK;
}
2

In case anyone else stumbles on this, my current solution is to directly manipulate the playback sound to insert data in the same way that I did the recording sound, via lock/unlock. So I removed the PCMREADCALLBACK and CREATESTREAM.

Now my problem is the playback is somewhat static/robotic sounding…

Updated code snippets.

        //Setup receive stream to playback sound
        exinfo2.cbsize = Marshal.SizeOf(typeof(FMOD.CREATESOUNDEXINFO));
        exinfo2.numchannels = NumOfChannels;
        exinfo2.format = soundFormat;
        exinfo2.defaultfrequency = SampleRate;
        exinfo2.length = exinfo.length;
        FMOD_ERRCHECK(RuntimeManager.CoreSystem.createSound(exinfo2.userdata, FMOD.MODE.LOOP_NORMAL | FMOD.MODE.OPENUSER, ref exinfo2, out recvSound));
        FMOD_ERRCHECK(RuntimeManager.CoreSystem.playSound(recvSound, channelGroup, true, out channel));

Loading data into playing channel:

void Update()
    {
        // Load playback buffer with data (Rx server)
        uint len1, len2;
        channel.setPaused(true);
        FMOD_ERRCHECK(channel.getPosition(out playbackPos, FMOD.TIMEUNIT.PCMBYTES));
        if (samplePos <= 0)
        {
            // Nothing new to play
            if (playbackPos < nextPlaybackPos)
            {
                // Already have data in buffer ahead to play, resume playing
                channel.setPaused(false);
            }
            return;
        }
        FMOD_ERRCHECK(recvSound.@lock(playbackPos, (uint)samplePos, out recvPtr1, out recvPtr2, out len1, out len2));
        if (len1 > 0)
        {
            Marshal.Copy(soundData, 0, recvPtr1, (int)len1);
            // shift whatevers left to the start
            shiftArrayStart(ref soundData, len1);
            samplePos -= (int)len1;
        }
        if (len2 > 0)
        {
            Marshal.Copy(soundData, 0, recvPtr2, (int)len2);
            // shift whatevers left to the start
            shiftArrayStart(ref soundData, len2);
            samplePos -= (int)len2;
        }
        nextPlaybackPos = playbackPos + len1 + len2;
        channel.setPaused(false);
        FMOD_ERRCHECK(recvSound.unlock(recvPtr1, recvPtr2, len1, len2));
    }

This seems to get latency about as low as possible.

3

You are correct, FMOD.MODE.CREATESTREAM does have a bit latency. I suspect the robotic sound is due to there being too little latency and the buffer not having enough recorded data to read. In the “record.cpp” Core API example there is 50ms of latency deliberately added to workaround this, and this gives a good balance of being an unnoticeable amount of latency without being so little that you get that robotic sound.

In the example this latency is achieved by waiting 50ms before calling playSound, You should be able to do something similar by waiting one update call before calling RuntimeManager.CoreSystem.playSound.

4

hello,codambro. how to set the samplePos default value and set back to default value when it’s modified?