I am trying to play raw samples pulled from a Theora codec in FMod. I know the Codec is reading the file correctly since video plays, and I can get FMod to read files, so seems I have gotten most things set up. This bit of code is where I have been struggling for a bit.
The attempt to use readData returns “FMOD_ERR_UNSUPPORTED (68).” But I have to admit, I am unsure if I am going about this wrong as well. After all this will get called time after time as the codec processes frames, and wondering if there is another way to go about it. In case it help’s the data in THEORAPLAY_AudioPacket is
typedef struct THEORAPLAY_AudioPacket
{
unsigned int playms; /* playback start time in milliseconds. */
int channels;
int freq;
int frames;
float *samples; /* frames * channels float32 samples. */
struct THEORAPLAY_AudioPacket *next;
} THEORAPLAY_AudioPacket;
Sound::readData isn’t quite the right function to call here, that’s for getting audio out of the Sound, not putting it in. You could use Sound::lock to get access to the internal pointers, write the sample data in, then call Sound::unlock to complete the operation.
Good to know, I saw in examples createSound should also be able to work with FMOD_OPENMEMORY, but when I try that I get “FMOD_ERR_FORMAT (19).” I have noticed there is just a handful of format options admittedly.
The data file coming in says “float samples; / frames * channels float32 samples. */” But both “FMOD_SOUND_FORMAT_BITSTREAM” and FMOD_SOUND_FORMAT_PCMFLOAT have not improved my luck.
Currently trying to see how to use lock and unlock though, feels a tad out of place but I might need to get this low level.
Yes, using FMOD_OPENMEMORY is a good option too, probably simpler for what you are doing. Just pass in FMOD_OPENRAW along with it since your data is raw PCM (this should fix the format error).
Certainly a partial fix! No more error, and audio is now trying to play, it’s not coming out right in the slightest, but I suspect that part might be on the decoders side. If I confirm that will mark your post as the solution, thank you!
Edit: Tweaking the flow of raw audio has let me get the correct sound, BUT it’s popping and not yet acceptable, gonna keep trying!
I’m curious, how long are these packets you are playing? If they are quite short you will hear gaps between each one.
If that’s the case a different approach is needed and depends largely on how you receive the audio / how much control you have.
Good timing! I was about to give a bit of an update, I now have it working %95 well, but not perfect. While I can now understand the audio coming though, it is rough at points and trying to smooth that out.
The sample itself is set up as “item->samples = (float *) malloc(sizeof (float) * frames * channels);” when first created, so just taking “sizeof (float) * frames * channels” for the FMOD_CREATESOUNDEXINFO length has solved most problems.
Your while loop is actually throwing audio away as it catches up, it only processes the most recent packet. while ((pAudio = THEORAPLAY_getAudio(pDecoder)) != NULL)
These packets will be quite small, so instead of playing each one individually (causing gaps between) you need to play it as a single stream.
Looking at the SDL example, I’d recommend copying their approach instead. In that example each packet is added to a queue, then the audio callback consumes from that queue as it plays. Take a look at the user_created_sound example we ship, this creates / plays a sound that fires a callback as it needs more data. Instead of generating a sine wave like our example, you would use the SDL example logic to consume from the audio queue. FMOD can directly ingest float audio, so no need to convert like the SDL example does, just set exinfo.format = FMOD_SOUND_FORMAT_PCMFLOAT.
Thanks for catching that, thought for some reason it was just throwing away stuff out of “date” (in case of lag.)
That said the callback doesn’t seem to be working right. Since I might be playing more then one video at a time I need either some kind of data storage or ID passed in so I can use the right queue.
Now I saw it looks like a Sound pointer is passed in, so I was trying to use its set/getUserData functions to pass in the queue. Problem is when using getUserData within the callback, it always returns null (doesn’t fail thankfully, but still null.) Stripped clean of error checking, here was what I made.
FMOD_RESULT F_CALLBACK pcmQueue(FMOD_SOUND* pSound, void* pData, unsigned int nDataLen)
{
FMOD::Sound* pCurrentSound = (FMOD::Sound*)pSound; //This refuses to work without type casting
void* pUserData = 0;
AudioCallbackQueue* pCurrent = 0;
The code you have shared looks like it would work, you just need to call Sound::setUserData on the Sound before you call System::playSound so the first callback has your user data ready to go.
pTest returns the expected value, so the sound is getting the data, just not handing over to the function. For the sake of certainty as well I can confirm I am using the latest (non-beta) version of FMOD API.
I spent a bit of time toying with FMOD_MODE settings but no luck. A bit of a tease since it feels like this is so close to working at last!
Oh I see, calling Sound::setUserData after System::createSound is too late, the callbacks are invoked as part of creating the sound, so when you get your first callback the user data hasn’t been set yet.
Instead of using the function, pass your user data in with fExtra.userdata.
That fixed it, and opened up a new issue, I swear we must be so very close! pcmreadcallback seems to only get called once.
If I just chain send the samples I get mostly working audio, but stuttering. So I changed it to give FMOD a linked list of samples, in hopes that fmod would use pcmreadcallback at it’s own leisure, and that would give each following sample as requested.
I suspect this is a problem with FMOD_CREATESOUNDEXINFO::Length? But unsure. I tried a number of things with it (including giving it the total size of the entire queue,) swapping createsound/createstream, and a wide number of mode setting with no luck. I got a slight improvement with FMOD_LOOP_NORMAL but it was mostly a wash. I did run a mp3 on the same groupchannel as a test so the rest of the system is working at least. As normal here is the latest codedump, callback function first.
FMOD_RESULT F_CALLBACK pcmQueue(FMOD_SOUND* pSound, void* pData, unsigned int nDataLen)
{
FMOD::Sound* pCurrentSound = (FMOD::Sound*)pSound;
void* pUserData = 0;
AudioCallbackQueue* pCurrent = 0;
FMOD_RESULT fError = FMOD_OK;
fError = pCurrentSound->getUserData( &pUserData );
pCurrent = (AudioCallbackQueue*)pUserData;
if(pCurrent != 0)
{
unsigned int nSize = sizeof(float) * (pCurrent->pRawAudio->frames) * (pCurrent->pRawAudio->channels);
if (nSize > nDataLen) //Not all samples are the same size.
nSize = nDataLen;
memcpy_s(pData, nDataLen, (pCurrent->pRawAudio->samples), nSize);
pCurrent = pCurrent->pNext;
}
pCurrentSound->setUserData(pCurrent);
return FMOD_RESULT::FMOD_OK;
}
void AudioSystem::RawAudio(RawSound* pAudio)
{
FMOD::Sound* pNewFSound = 0;
FMOD_RESULT resError;
SoundHandle* pCurrentHandle = mapSoundObjects[pAudio->nID];
unsigned int nSampleSize = 0;
FMOD_CREATESOUNDEXINFO fExtra;
if (pCurrentHandle->pChannel != 0) //Checking we don't steal a channel by mistake
{
pCurrentHandle->pChannel->getCurrentSound(&pNewFSound);
if (pNewFSound == pCurrentHandle->pSound)
pCurrentHandle->pChannel->stop(); //If we reached here, we where still using the channel, so we can tell it to stop.
}
while (pCurrentHandle->pQueueStart != 0) //Old queue out of date, clean up.
{
AudioCallbackQueue* pNext = pCurrentHandle->pQueueStart->pNext;
delete pCurrentHandle->pQueueStart;
pCurrentHandle->pQueueStart = pNext;
}
pCurrentHandle->pQueueStart = pAudio->pQueue;
std::cout << "Giving Audio Queue \n"; //Lets me see in a console window the rate of function calls
nSampleSize = (sizeof(float) * (pCurrentHandle->pQueueStart->pRawAudio->frames) * (pCurrentHandle->pQueueStart->pRawAudio->channels));
memset(&fExtra, 0, sizeof(FMOD_CREATESOUNDEXINFO));
fExtra.length = nSampleSize;
fExtra.cbsize = sizeof(FMOD_CREATESOUNDEXINFO);
fExtra.defaultfrequency = pCurrentHandle->pQueueStart->pRawAudio->freq;
fExtra.numchannels = pCurrentHandle->pQueueStart->pRawAudio->channels;
fExtra.userdata = (void*)(pCurrentHandle->pQueueStart);
fExtra.format = FMOD_SOUND_FORMAT_PCMFLOAT;
fExtra.pcmreadcallback = pcmQueue;
resError = m_pSystem->createStream(0, FMOD_CREATESTREAM | FMOD_OPENUSER, &fExtra, &pNewFSound);
resError = m_pSystem->playSound(pNewFSound, vecSoundsAndChannels[ENUMSOUNDTYPE::eVIDEOAUDIO]->pChannelGroupData, false, &(pCurrentHandle->pChannel));
}
Sorry for messy code, doing this in the middle of a game jam so lots of panic!
If you want the read callback to keep on firing in realtime as it plays you will want to use FMOD::createStream and play only a single Sound, continuing to feed it as the callbacks come in. Setting the fExtra.length will control how long the overall stream is so it can naturally stop at the right time, assuming you have the length of the entire video? Whether you use looping comes down to what you want to happen when playback reaches the length you’ve provided, I’d ignore that for now until you have the whole stream playing once nicely.
Also you can use fExtra.decodebuffersize to control the size of the requests in the callback.
You’ll need some more logic in your callback if the callback requests a chunk smaller or larger than the current queue item provides. If FMOD requests more you’ll need to consume more than one queue item, if FMOD requests less you’ll need to remember where you are up to in the current queue item, so the next request consumes the remainder.
That seems to have moved things much the right direction! Fmod’s call back is happening now constantly, asking for large packets in the call back so I need to spend a little time correctly feeding it, but hopefully that will solve it.
The jam is over so this is taking place during spare time now, but I greatly wanna see this work so not going to give up yet, updates just might be slow.
In the hopeful chance everything gets working soon enough I’ll make a final post and mark the subject as closed. This seems so close though!