Android 3D spatialisation + DSP processing

FMOD Engine
,
1

Spatialisation with android + head tracking on Pixel Pro devices with Pixel Buds Pro is working (plus a bunch of other headphones that support head tracking, e.g. Redmi Buds Pro, SHOKZ OpenFit Pro etc.). It head tracks and spatialises correctly, and is quite impressive!

I’ve hit a separate issue though, which is that I also use a custom DSP to process ambisonics 1st, 2nd and 3rd order files and output the decoded version. Previously I was doing this using the Binauralizer, which outputs stereo, and I piped in the rotation of the headtracking directly, but since now the android way to do it is to output 7.1.4 (atmos) directly to the system it requires me to decode the ambisonics to 7.1.4 instead… which is entirely possible – I can just configure a 7.1.4 speaker array using the right coefficients.

The issue is that the DSP callback returns e.g. 4 channels for input (which is correct for the ambisonic 1st order file) and 4 channels for output (which is not correct)

inline FMOD_RESULT F_CALL ambisonicDSPCallback(FMOD_DSP_STATE *dsp_state, float *inbuffer, float *outbuffer, unsigned int length, int inchannels, int *outchannels) {

and in the DSP creation callback it says that the speakermode is 7.1.4, but the speakermodeoutput is stereo? Which means I can’t write the decoded 12 channels in the DSP callback function.

inline FMOD_RESULT F_CALL ambisonicDSPCreateCallback(FMOD_DSP_STATE *dsp_state)
{
    unsigned int blocksize;
    FMOD_SPEAKERMODE speakermode;
    FMOD_SPEAKERMODE speakermodeoutput;
    unsigned int numspeakers = 2;
    FMOD_RESULT result;
    
    result = dsp_state->functions->getspeakermode(dsp_state, &speakermode, &speakermodeoutput);
    if (result != FMOD_OK) {
        LOGE("fmod:: Error getting speaker mode because: %s\n", FMOD_ErrorString(result));
    }

Should I be setting the dsp_state→speakermode or the dsp_state→channelmask?

2

DSPs can define the output channels count by setting buffernumchannels and speakermode during the FMOD_DSP_PROCESS_QUERY stage of FMOD_DSP_PROCESS_CALLBACK:

FMOD_RESULT F_CALL dspprocess(FMOD_DSP_STATE *dsp_state, unsigned int length, const FMOD_DSP_BUFFER_ARRAY *inbufferarray, FMOD_DSP_BUFFER_ARRAY *outbufferarray, FMOD_BOOL inputsidle, FMOD_DSP_PROCESS_OPERATION op)
{
    FMODGainState *state = (FMODGainState *)dsp_state->plugindata;

    if (op == FMOD_DSP_PROCESS_QUERY)
    {
        if (outbufferarray && inbufferarray)
        {
            // Always output Atmos
            outbufferarray[0].buffernumchannels[0] = 12;
            outbufferarray[0].speakermode = FMOD_SPEAKERMODE_7POINT1POINT4;
        }

        if (inputsidle)
        {
            return FMOD_ERR_DSP_DONTPROCESS;
        }
    }
    else
    {
        // Do what you want with the input and mix it to your output channel count
        state->read(inbufferarray[0].buffers[0], outbufferarray[0].buffers[0], length, inbufferarray[0].buffernumchannels[0]);
    }

    return FMOD_OK;
}

I think that would be the only missing step, please let me know if that still doesn’t work!

3

I have had some success. Your approach works, setting the buffernumchannels and speakermode, which is fantastic. The DSP processes audio, but it doesn’t sound correct.

When I call
result = dsp_state->functions->getspeakermode(dsp_state, &speakermode, &speakermodeoutput);

in the create callback, the speakermodeoutputis set to stereo (although the speakermode is correctly set to FMOD_SPEAKERMODE_7POINT1POINT4).

Does that mean the channel itself is set to stereo? In which case I’d really like to set the output directly to 7.1.4 and have that as raw output to the android system (which will then spatialise it itself).

I tried setting the channel format for the DSP, but it just crashes if I do that:
result = ambisonic_dsp->setChannelFormat(FMOD_CHANNELMASK_7POINT1, 12, FMOD_SPEAKERMODE_7POINT1POINT4);

4

The channel doesn’t really have the concept of a speakermodeoutput on its own. The channel is a chain of DSPs, and the last DSP (dsp tail) in it’s chain is essentially the channel’s output.

You’ll want to make sure anything downstream of the DSP has the correct speaker mode so that it doesn’t get downmixed, and you’ll want to ensure FMOD is initialized with FMOD_SPEAKERMODE_7POINT1POINT4 (System::setSoftwareFormat). If you initialize with FMOD_INIT_PROFILE_METER_ALL you’ll be to connect the Core API Profiler Tool and see if there is anything collapsing the speaker mode. For example:

image
image1071×619 22 KB

Custom up-mix example 
    FMOD_DSP_DESCRIPTION desc {};
    strcpy(desc.name, "Custom up-mix DSP");
    desc.numinputbuffers = 1;
    desc.numoutputbuffers = 1;
    desc.process = [](FMOD_DSP_STATE *dsp_state, unsigned int length, const FMOD_DSP_BUFFER_ARRAY *inbufferarray,
                       FMOD_DSP_BUFFER_ARRAY *outbufferarray, FMOD_BOOL inputsidle, FMOD_DSP_PROCESS_OPERATION op)
    {
        if (op == FMOD_DSP_PROCESS_QUERY)
        {
            if (outbufferarray && inbufferarray)
            {
                // Always output Atmos
                outbufferarray[0].buffernumchannels[0] = 12;
                outbufferarray[0].speakermode = FMOD_SPEAKERMODE_7POINT1POINT4;
            }

            if (inputsidle)
            {
                return FMOD_ERR_DSP_DONTPROCESS;
            }
            return FMOD_OK;
        }

        memset(outbufferarray->buffers[0], 0.0f, length * outbufferarray->buffernumchannels[0] * sizeof(float));

        for (int i = 0; i < inbufferarray->buffernumchannels[0]; i++)
        {
            for (int j = 0; j < length; j++)
            {
                for (int k = 0; k < outbufferarray->buffernumchannels[0]; k++)
                {
                    //Give everything to everyone
                    outbufferarray->buffers[0][k + j * outbufferarray->buffernumchannels[0]] +=
                        inbufferarray->buffers[0][i + j * inbufferarray->buffernumchannels[0]] /
                        inbufferarray->buffernumchannels[0];
                }
            }
        }

        return FMOD_OK;
    };

    FMOD::DSP *dsp;
    unsigned int handle {};
    system->registerDSP(&desc, &handle);
    system->createDSPByPlugin(handle, &dsp);
    channel->addDSP(FMOD_CHANNELCONTROL_DSP_HEAD, dsp);