Integrating with head tracking on android

FMOD Engine
1

The latest versions of android (13+) support head tracking. The architecture is described here: 空間音頻和頭部跟踪  |  Android Open Source Project

From the diagrams on that page, it should be as simple as outputting 5.1 / 7.1.4 from the app and making sure that head tracking is enabled (on a compatible device, such as the Google Pixel Buds Pros).

They provide callbacks for checking spatialisation support here: Spatializer  |  Android Developers

and there’s a general description for devs here: Spatial Audio  |  Android media  |  Android Developers

Has anyone tried/succeeded/failed with this? Are there any problems, such as with the output of fmod in the signal chain described above?

2

I don’t believe that there is anything special needed from FMOD, just the usual spatial audio workflow should be enough as the positioning is all handled on the game engine side.

3

Hi,

I have also been trying to get this working with Pixel Bud Pros and a pixel 8.
Running on Android the driver returned by getDriverInfo is “Android audio output” and it’s stereo at 48kHz. That matches the documentation as the expected default driver.

The software mixer is set to FMOD_SPEAKERMODE_SURROUND and I can see that there’s a mixdown taking place:

2024-06-20 11:33:34.104 10422-10422 AudioEngine             com.kersnazzle.soundscapealpha       D  27 of ../../src/fmod_downmix.cpp: dspbuffersize: 512  rate: 48000. output speaker mode: 3, input speaker mode 5

Because I have spatialization enabled on the headphones I would expect it to be possible to send them multi-channel audio and then the Android spatializer could do its thing. If only stereo audio is sent, then the spatializer won’t do anything.

Perhaps I should be calling System::setOutput() but there doesn’t seem to be an FMOD_OUTPUTTYPE appropriate on Android that isn’t FMOD_OUTPUTTYPE_AAUDIO.

Any suggestions gratefully received,

Many Thanks

Dave Craig

4

Did you try setting to FMOD_SPEAKERMODE_7POINT1POINT4 with System::setSoftwareFormat()? I read somewhere in the android documentation that that’s the highest spatialisation setup supported.

Also, did you check in the android code that all the spatialisation setup is correct? (checklist below)

Does the device support spatialization? getImmersiveAudioLevel() is not SPATIALIZER_IMMERSIVE_LEVEL_NONE
Is spatialization available?
Availability depends on compatibility with the current audio output routing. isAvailable() is true
Is spatialization enabled? isEnabled() is true
Can an audio track with the given parameters be spatialized? canBeSpatialized() is true

There’s also an interesting bit about setting the max number of channels on android: Spatial Audio  |  Android media  |  Android Developers

5

Yes, that made no difference. It still results in a mixdown taking place, just from a different surround format.

Yes I had done this too, it’s all setup and spatialization is available. I think that the underlying issue is that the FMOD code needs to do these checks and then based on the result it should be setting the audio device output format to a surround format and not stereo. That will allow the Android Spatializer to then do the mixdown to the headphones.
From my application’s perspective, it will need to stop controlling the FMOD listener direction if head tracking and spatialization is active (it currently uses the orientation of the phone). I’m assuming that the FMOD surround “stage” would be held constant e.g. with the L C R speakers on the “north wall”, and then the Spatializer head tracking would then deal with the listener’s orientation.
Hopefully that all makes sense! For background, the application here is for an Android port of a navigation app for the visually impaired.

Thanks

Dave.

6

Thanks for all the information, we are currently investigating this and will get back to you shortly.

7

Yes I had done this too, it’s all setup and spatialization is available. I think that the underlying issue is that the FMOD code needs to do these checks and then based on the result it should be setting the audio device output format to a surround format and not stereo. That will allow the Android Spatializer to then do the mixdown to the headphones.

@davecraig My guess is that fmod is detecting internally it’s on android and always mixing down to stereo regardless of what we set in setSoftwareFormat() – i.e. doing its own spatialization. Now it needs to respect what we set (i.e. FMOD_SPEAKERMODE_7POINT1POINT4) and just pass it to the internal engine.

@cameron-fmod the audio spatialisation workflow changed in android 13 - see here: 서라운드 오디오 및 머리 추적  |  Android 오픈소스 프로젝트  |  Android Open Source Project

You’d need to check that the current android SDK version is >= 33, getting the android SDK version in the NDK, then allow setting other speaker setups.

This is from SO here.

#include <cutils/properties.h>

// ...

int myfunction() {
    char sdk_ver_str[PROPERTY_VALUE_MAX] = "0";
    property_get("ro.build.version.sdk", sdk_ver_str, "0");
    sdk_ver = atoi(sdk_ver_str);
    // ...   
}

Sorry if this is all obvious, just thought I’d paste here in case it helps.

Thanks so much! I feel like it’s getting closer.

1 Like
8

Thank you for your patience, it looks like we will need to update our API to allow for Android Spatial Audio. I have raised this as a task to fix in an upcoming release.

1 Like
9

Amazing, thank you @cameron-fmod :clap: – keep us updated and let me know if you need any more info from us, we’ve been looking at this in some depth.