Android 3D spatialisation + DSP processing

FMOD Engine
,
1

Spatialisation with android + head tracking on Pixel Pro devices with Pixel Buds Pro is working (plus a bunch of other headphones that support head tracking, e.g. Redmi Buds Pro, SHOKZ OpenFit Pro etc.). It head tracks and spatialises correctly, and is quite impressive!

I’ve hit a separate issue though, which is that I also use a custom DSP to process ambisonics 1st, 2nd and 3rd order files and output the decoded version. Previously I was doing this using the Binauralizer, which outputs stereo, and I piped in the rotation of the headtracking directly, but since now the android way to do it is to output 7.1.4 (atmos) directly to the system it requires me to decode the ambisonics to 7.1.4 instead… which is entirely possible – I can just configure a 7.1.4 speaker array using the right coefficients.

The issue is that the DSP callback returns e.g. 4 channels for input (which is correct for the ambisonic 1st order file) and 4 channels for output (which is not correct)

inline FMOD_RESULT F_CALL ambisonicDSPCallback(FMOD_DSP_STATE *dsp_state, float *inbuffer, float *outbuffer, unsigned int length, int inchannels, int *outchannels) {

and in the DSP creation callback it says that the speakermode is 7.1.4, but the speakermodeoutput is stereo? Which means I can’t write the decoded 12 channels in the DSP callback function.

inline FMOD_RESULT F_CALL ambisonicDSPCreateCallback(FMOD_DSP_STATE *dsp_state)
{
    unsigned int blocksize;
    FMOD_SPEAKERMODE speakermode;
    FMOD_SPEAKERMODE speakermodeoutput;
    unsigned int numspeakers = 2;
    FMOD_RESULT result;
    
    result = dsp_state->functions->getspeakermode(dsp_state, &speakermode, &speakermodeoutput);
    if (result != FMOD_OK) {
        LOGE("fmod:: Error getting speaker mode because: %s\n", FMOD_ErrorString(result));
    }

Should I be setting the dsp_state→speakermode or the dsp_state→channelmask?