Realtime playback from a device

FMOD Engine
1

Hello,

I am trying to capture the output from a device and play it in real time through a FMOD Sound (FMOD_OPENUSER | FMOD_LOOP_NORMAL | FMOD_CREATESTREAM). Minimizing the delay between capture and playback here is important.

The issues arise when FMOD requests more data than I can fetch from the device. Should I keep looping or is there a return value to the pcmreadcallback that will tell fmod to request this packet later? Is there a way to partially fill the buffer with what we can and let FMOD know the number of bytes written ?

As I could only find little documentation on the behavior of the read callback, especially when it comes to valid return values, nor could I find equivalent examples.

My current approach does the following:

  • decodebuffersize has been set to 2048 as this was the minimum value I observed to be playing without stuttering
  • Read callback keeps looping until the requested amount of data is written (could stall the thread longer than necessary for this even though it has partially filled the data)
  • If more data is fetched from the device than needed, saves it in a temporary buffer for the next frame

Is this how FMOD deals with devices internally? I need to play the captured sound in real time and without distortion or stuttering, this is very important. I would love to see how FMOD deals with recording the microphone input, for instance :wink:

Pseudocode:

char savedDataBuffer[BIG_VALUE];
uint savedDataLength = 0;

FMOD_RESULT readCallback(FMOD_SOUND *sound, char* data, uint len)
{
	if(savedDataLength)
	{
		memcpy(data, savedDataBuffer, savedDataLength);
		savedDataLength = 0;
		len -= savedDataLength;
	}
	
	while(len > 0)
	{
		while(device.hasDataToCapture())
		{
			uint writeLength = min(device.captureSize(), len);
			
			memcpy(data, device.captureBuffer(), writeLength);
			len -= writeLength ;
            			
			if(device.captureSize() > writeLength)
			{
				uint remainingLength = device.captureSize() - writeLength;
				memcpy(savedDataBuffer, device.captureBuffer() + writeLength, remainingLength);
				savedDataLength = remainingLength;
			}
		}
     }
     return FMOD_OK;
 }
1 Like
2

Bump, I am truly blocked in my progress by this. Any help will be appreciated.

3

FMOD requests more data than I can fetch from the device

The only thing you can do there is stall inside the read callback until the data is ready. This is just the same if there was a network stall or packet loss, or the media is slow (like a cdrom).

If the decode buffer size is big enough then it won’t stutter. You wouldn’t be making it smaller , you’d be making it bigger, to hid any glitches or stalls.
It would only stutter if the length of the delay getting the data was longer than the time it takes to play a stream decode buffer.

4

Considering i am playing the sound in real time, if the playback has, say 50 ms delay, but fmod requests ~400ms (default value per documentation) every time, then the playback will be glitching every single time, as it will take at least 400 ms of real time to gather 400ms of buffer.

5

streams are not for ‘real time’, their purpose is to allow large amounts of data to be buffered up so you can hide the slowness of the media, netstreams and cdroms for example.

If you want low latency access directly to the mix stream, add a dsp callback instead. you can either use addDSP to add it to a channelgroup, or even have it play on a channel (and get the features of a channel like being able to set the frequency of it) using System::playDSP

if FMOD wants 5ms of data in this case you just buffer your own reads somewhere else and feed from that buffer in the dsp callback.

6

Thank you, I will try this approach. The result I am trying to obtain is the same as what you get when recording from e.g. the microphone using System::recordStart.

7

Anwsering my own post, as I am now getting satisfactory results using another approach.

The best solution I found in the end was to create a looping sound with a set buffer, not using FMOD_CREATESTREAM.

I then have my capture thread (or main thread) use lock/unlock to write into the buffer and control the cursor of the playback to ensure a stable latency.