I just checked FMOD_OUTPUTTYPE constants and saw FMOD_OUTPUTTYPE_AAUDIO!
Finally AAudio support on Android!
I’m very happy now and really appreciate your effort.
By the way, I want to know if there’s any new documentation related to FMOD’s AAudio or low latency on Android,
something I should know, any tips or advices.
I’m curious because your documentations here helped when achieving low latency on Android devices with the old FMOD Studio API.
There isn’t really anything that is required or different with AAudio. The documentation in general, including the latency parts will still apply.
We default to it with api 26 (android 8.0 aka Oreo). Otherwise if the Low latency flag is set we default to OpenSL, otherwise we fall back to Audiotrack.
Big thanks!
You mean, even if I’m only targeting Android 8.0 and up, there’s no magical thing in AAudio so I still need to configure the settings exactly the same way as OpenSL ES, right?
It’s been a long time since I last used FMOD Studio on Android, so I had to read all the documents over again, and this time, I’m a little confused.
To be honest, VERY CONFUSED!
From one of your documents about Android Audio Latency, it says :
Devices which report FEATURE_AUDIO_LOW_LATENCY will be able to achieve lower latency playback, especially if the below tips are followed.
For API level 17 devices using the OpenSL output mode you can achieve lower latency by using System::getDriverInfo to fetch the recommended sample rate
and applying the value to System::setSoftwareFormat, by default the sample rate is 24KHz to keep CPU overheads low, using a higher rate will cost more CPU time.
“by default the sample rate is 24KHz to keep CPU overheads low”
Yeah, this one was what I referred to when I last wrote my android app using FMOD Studio.
But this time I found this too.
In the page, it says :
It is also highly recommended that you initialize the FMOD Java component, this will allow loading assets from the APK and automatic configuration for lowest latency.
This explanation was about calling org.fmod.FMOD.init(this) on the Java side.
In fact, I did this too without thinking too much last time, because I had to call it for assets in the APK anyway.
But I also found this page :
Core API - Android - Improved latency by automatically resampling to the native rate to enable the fast mixer.
This seems to be about org.fmod.FMOD.init() too, so now I’m in doubt if above manual settings are still required.
Because I was so curious, I decompiled the fmod.jar file, but the init() method did nothing but set a static Context variable.
There were also some helper methods such as supportsLowLatency(), getOutputSampleRate() and getOutputBlockSize(),
(which check FEATURE_AUDIO_PRO, FEATURE_AUDIO_LOW_LATENCY flags and PROPERTY_OUTPUT_SAMPLE_RATE, PROPERTY_OUTPUT_FRAMES_PER_BUFFER properties…)
so I’m just assuming FMOD would configure itself on the native side when the Context variable is set by init().
Also, reading the first quote again, I’m getting confused even more.
Most smartphones’ sample rates these days are either 44.1KHz or 48KHz, meaning more CPU time according to your explanation.
I personally use a lot of DSPs like FMOD_DSP_TYPE_SFXREVERB and FMOD_DSP_TYPE_PITCHSHIFT, and I even call FMOD_Channel_SetFrequency().
In your documentation somewhere, I also saw resampling can’t be avoided when DSPs are used.
So I don’t think it’s that beneficial to increase CPU time in order to enable the fast mixer. But, well, I’m just guessing here.
On the other hand, what if I keep using your default value 24000 but my source files are recorded in 48000 or 44100.
Resampling all those files may require more CPU time in this case.
Maybe resampling is less expensive than processing high sample rate audios, and all these are just a balance between trade-offs problem?
Because I have no idea at all how FMOD Studio works internally, I’m also guessing this.
System::setSoftwareFormat() may set OpenSL ES’s SLDataFormat_PCM.samplesPerSec value ONLY.
Or it only sets the FMOD API’s configuration for internal processing. (And then resample again to match OpenSL ES’s right before sending the audio?)
Or both.
FMOD DSP network is one of the important aspects in FMOD, right?
In this link, it says :
Wavetable Unit : This unit reads raw PCM data from the sound buffer and resamples it to the same rate as the soundcard. A Wavetable Unit is only connected when the user calls System::playSound.
Once resampled, the audio data is then processed (or flows) at the rate of the soundcard. This is usually 48khz by default. (22khz on iOS)
This confuses me again about two things.
“the same rate as the soundcard”? Did you want to say the same rate as the value set by System::setSoftwareFormat, didn’t you?
If not, it’s very confusing and I need an explanation on this.
Also it’s unclear if it will resample just once when System::playSound() is first called and keep it, or will resample everytime the audio data is processed during the playback.
As for the sound created with FMOD_CREATESTREAM or FMOD_CREATECOMPRESSEDSAMPLE which needs DSPCodec Unit’s help, the latter makes sense, but as for FMOD_CREATESAMPLE, I’m not sure about that.
When FMOD_System_CreateSound() is called with the FMOD_CREATESAMPLE flag, I know the sound buffer will contain uncompressed PCM data.
But what about the sample rate of the sound?
Sorry for spamming questions like this but because high latency really matters on my app, I need to clear these things up before starting the project.
You’ve managed to figure most of it out, even though some of the docs do seem to be a bit misleading. I have made a couple of tasks to fix up some of the docs, to make them a bit clearer and fix up any wording.
The steps documented to get / set the native output rate and buffer size (while not harmful) are not necessary anymore. We do this internally as long as the user calls org.fmod.FMOD.init(), as you discovered.
There are performance benefits to running the mixer at 24KHz and latency benefits running at 48KHz. This is why we handle it internally now, we run the mixer at 24KHz, then resample to 48KHz for the output to get low latency (best of both worlds) with no user configuration required.
We resample from the source file sample rate to the system rate on playback. Then we resample from the system rate to the soundcard rate (opensl / aaudio) on the final submix.
Using create sample or stream only affects the decoding of compressed audio. Even with create sample the PCM buffer produced is in source sample rate, it will be resampled to the system rate as it plays (taking into account calls to setPitch).
Wow! After reading your reply, now I can better understand how FMOD Studio works. Thank you very much.
I think I’m almost there now.
Still lots of questions below, but this time it’ll be easier for you to answer, I guess.
Also, this is the last time. I promise.
Q1.
In this page, it says :
“Be aware that this will only happen if there are no pitch / frequency settings applied to the Channel, so this trick is often good for music.”
Why are pitch and frequency specifically mentioned here?
Is it because the two are the only processing which requires resampling under the hood, unlike other DSPs?
Q2.
As you know, in order to achieve low-latency on Android, it’s also important to set OpenSL ES’ buffer size properly using a value returned by getProperty(AudioManager.PROPERTY_OUTPUT_FRAMES_PER_BUFFER).
Now I know fmod.jar’s init() will adjust the sample rate automatically.
But what about buffer size? Will init() take care of it as well?
Q3.
It’s somewhat obscure whether FMOD_System_SetDSPBufferSize() sets FMOD System’s internal buffer or OpenSL ES’ (or AAudio’s).
I once wrote a Windows application with FMOD which used ASIO.
At that time, it crashed when unsupported values by my audio interface were set.
Also, one poster’s question here says only values supported by the audio interface worked.
But your document says
“Sets the buffer size settings for the FMOD software mixing engine.”
So this is confusing.
Is FMOD_System_SetDSPBufferSize() for the FMOD System only?
If yes, how can I set the OpenSL ES’ (or AAudio’s) buffer size?
Even aside from latency issue, setting proper buffersize is important because this accepted answer from an Android audio team developer says there’s a bug on Android M :
“Make your buffer size a multiple of AudioManager.getProperty(PROPERTY_OUTPUT_FRAMES_PER_BUFFER). Otherwise your callback will occasionally get two calls per timeslice rather than one.
… (On Android M, it is very important to use EXACTLY the system buffer size, due to a bug in the buffer handling code.)”
Last question.
This is about SAMPLE RATE and FMOD_System_SetSoftwareFormat(). again.
Writing separately because it needs some long explanation.
This says :
“Sets the output format for the software mixer.”
According to this, FMOD_System_SetSoftwareFormat() is for OUTPUT.
And this says :
“using the OpenSL output mode you can achieve lower latency by using System::getDriverInfo to fetch the recommended sample rate and applying the value to System::setSoftwareFormat, by default the sample rate is 24KHz to keep CPU overheads low, using a higher rate will cost more CPU time.”
So, according to this,
even though it’s 24000 by default to reduce CPU cost on Android, it is required to set a proper sample rate value using FMOD_System_SetSoftwareFormat() to achieve low-latency.
But this says :
“Core API - Android - Improved latency by automatically resampling to the native rate to enable the fast mixer.”
So, now FMOD Studio does something cool automatically under the hood for us.
And according to your comment,
fmod.jar’s init() does the job, so we don’t need to call FMOD_System_SetSoftwareFormat() ourselves to match the sample rate anymore.
Considering all of the above and some other useful info from you, my conclusion is :
a. FMOD Studio has two separate sample rates. One for internal processing (mixer) and one for output (final submix).
b. FMOD_System_SetSoftwareFormat() configures the output sample rate.
c. For low-latency audio on Android, we need to match sample rates between the device and FMOD System calling FMOD_System_SetSoftwareFormat().
d. But fmod.jar’s init() indirectly makes a call to FMOD_System_SetSoftwareFormat() with proper values instead of us now.
Cool!
HOWEVER this says something different, I think.
"There are two sample rates you need to think about when optimizing, the System rate and the source audio rate.
You can control the System sample rate by using System::setSoftwareFormat (sampleRate, …), which by default is 48KHz. Reducing this can give some big wins in performance because less data is being produced. This setting is a trade off between performance and quality.
To control the source audio rate you can resample using your favorite audio editor or use the sample rate settings when compressing using the FSBank tool or the FSBankLib API. All audio will be sent to a resampler when it is played at runtime, if the source sample rate and the System rate match then the resampler can be essentially skipped saving CPU time. Be aware that this will only happen if there are no pitch / frequency settings applied to the Channel, so this trick is often good for music."
Above explains as if FMOD_System_SetSoftwareFormat() has something to do with the internal processing.
Also this says :
“Wavetable Unit : This unit reads raw PCM data from the sound buffer and resamples it to the same rate as the soundcard. A Wavetable Unit is only connected when the user calls System::playSound. Once resampled, the audio data is then processed (or flows) at the rate of the soundcard. This is usually 48khz by default. (22khz on iOS)”
Reading all these, it seems like FMOD_System_SetSoftwareFormat() controls both the output and internal (mixer) sample rate.
But this conflicts with your explanation, that internally 24000 is used and then will be resampled to the device’s sample rate before outputting.
So this is my last question :
Could you correct me where I’m wrong here?
If FMOD_System_SetSoftwareFormat() sets the output sample rate only, then I want to know how to change the mixer sample rate too.
FMOD team’s decision is correct most of the time, but some projects need high-quality audio on mobile.
I personally prepared audio sources recorded in 48kHz for a long time for my project, so really want to keep the quality.
The short answer is that you now just need to call the fmod.jar’s init() and the rest will be taken care of for you.
Q1. Why are pitch and frequency specifically mentioned here?
Is it because the two are the only processing which requires resampling under the hood, unlike other DSPs?
This resampling is referring to per source file, whereas previously we have been talking more about resampling the mixer (the final buffer sent to the hardware). If the pitch or frequency are modified then that will effect the sample rate, requiring the use of the resampler, while other effects don’t effect the sample rate.
Q2. Now I know fmod.jar’s init() will adjust the sample rate automatically.
But what about buffer size? Will init() take care of it as well?
The jar files Init() doesn’t set the sample rate or buffer size, it is required to allow FMOD to get the hardware sample rate and buffer sizes. This is what FMOD then uses to convert the final mixer buffer to when it passes it to the output/hardware.
Q3. It’s somewhat obscure whether FMOD_System_SetDSPBufferSize() sets FMOD System’s internal buffer or OpenSL ES’ (or AAudio’s).
FMOD_System_SetDSPBufferSize and FMOD_System_SetSoftwareFormat sets FMOD’s internal values, we make sure it is in the correct buffer size and sample rate to be passed to the hardware.
We do this to ensure tht you get access to the low latency feature.
Q4. If FMOD_System_SetSoftwareFormat() sets the output sample rate only, then I want to know how to change the mixer sample rate too.
FMOD_System_SetSoftwareFormat sets the mixer sample rate, the output sample rate is determined by the hardware.
Thanks for your great explanation and confirmation.
I think now I better understand FMOD Studio API.
To be honest, it may be just me, but your manual seems to confuse a little bit.
As for your third answer “FMOD_System_SetDSPBufferSize and FMOD_System_SetSoftwareFormat sets FMOD’s internal values”,
I’m still not 100% convinced because your old manual said about some hardware specific settings.
"Platform Notes
Some output modes (such as FMOD_OUTPUTTYPE_ASIO) will change the buffer size to match their own internal optimal buffer size. Use System::getDSPBufferSize after calling System::init to see if this is the case.
Xbox 360 defaults to 256 sample buffersize and 4 for numblocks. This gives a 5.333ms granularity with roughly a 10-15ms latency.
PS3 ignores this function. Check FMOD_PS3_EXTRADRIVERDATA to control output latency."
If FMOD_System_SetDSPBufferSize() only has something to do with FMOD’s internal settings but not hardware, then why does a certain platform have to ignore the function call?
Also it doesn’t explain why my application had to crash with wrong values either…
Anyway, thank you very much.
Have a great weekend! 
Some of our legacy output plugins had caveats, this isn’t an issue anymore.