C# unable to ear spatialiced audio using coreapi and resonanceaudio

Hello guys,

I’m trying to build a small working example using monogame, fmod with coreapi and resonanceaudio
in C# to get play one audio in 3d and with HRTF, but.

I got start all things fine. I created a small assembly with the c# fmod headers (to carry all of them in a dll) and the rest of things are in the game Class of the project.

I can without problems load the engine, load the resonanse pluygin, create the DSP and load and play the sound, but the sound ears without the spatialized effect; I based my code on a previous example that I saw on this forum, but it was in c++ and from middle 2018.

I don’t know if the resonance audio properties or similar things had been changed to this date; I can’t find any documentation about the parameters of resonance plugin, or which is the correct way to use it with core (or lowlevel api. All the documentation talks about the studio version.

Well, this is the code of my main class:

Can you detect something bad?

Thanks for your help :slight_smile:

using Microsoft.Xna.Framework;
using Microsoft.Xna.Framework.Graphics;
using Microsoft.Xna.Framework.Input;
using FMOD;
using System;

namespace game1_with_fmod_wrapper
{
    /// <summary>
    /// This is the main type for your game.
    /// </summary>
    public class Game1 : Game
    {
        GraphicsDeviceManager graphics;
        SpriteBatch spriteBatch;
        FMOD.System fmod;
        ChannelGroup masterChannel;
        ChannelGroup world3DChannel;
        uint raHandle;
        uint ralistenerHandle;
        uint raSourceHandle;
        DSP listenerDSP;
        VECTOR listenerPos = new VECTOR { x = 10, y = 0, z = -5 };
        VECTOR listenerVel = new VECTOR { x = 0, y = 0, z = 0 };
        VECTOR listenerForward = new VECTOR { x = 0, y = 0, z = 1 };
        VECTOR listenerUp = new VECTOR { x = 0, y = 1, z = 0 };

        public Game1()
        {
            graphics = new GraphicsDeviceManager(this);
            Content.RootDirectory = "Content";
        }

        /// <summary>
        /// Allows the game to perform any initialization it needs to before starting to run.
        /// This is where it can query for any required services and load any non-graphic
        /// related content.  Calling base.Initialize will enumerate through any components
        /// and initialize them as well.
        /// </summary>
        protected override void Initialize()
        {
            // TODO: Add your initialization logic here
            FMOD.Factory.System_Create(out fmod);
            fmod.init(1024, INITFLAGS.NORMAL,(IntPtr)OUTPUTTYPE.AUTODETECT);
            fmod.getMasterChannelGroup(out masterChannel);
            fmod.createChannelGroup("world", out world3DChannel);
            FMOD.RESULT ldpluginresult = fmod.loadPlugin("plugins/resonanceaudio.dll", out raHandle);

            fmod.getNestedPlugin(raHandle, 0, out ralistenerHandle);
            fmod.getNestedPlugin(raHandle, 2, out raSourceHandle);
            fmod.createDSPByPlugin(ralistenerHandle, out listenerDSP);
            world3DChannel.addDSP(FMOD.CHANNELCONTROL_DSP_INDEX.TAIL,listenerDSP);
            fmod.set3DNumListeners(1);
            fmod.set3DListenerAttributes(0,ref listenerPos, ref listenerVel, ref listenerForward, ref listenerUp);
            

            base.Initialize();
        }

        /// <summary>
        /// LoadContent will be called once per game and is the place to load
        /// all of your content.
        /// </summary>
        protected override void LoadContent()
        {
            // Create a new SpriteBatch, which can be used to draw textures.
            spriteBatch = new SpriteBatch(GraphicsDevice);
            Sound jumpsound;
            fmod.createSound("grito.wav",  MODE._3D| MODE.LOOP_NORMAL, out jumpsound);
            Channel jumpchannel;
            fmod.playSound(jumpsound, world3DChannel,paused: true,channel: out jumpchannel);
            fmod.createDSPByPlugin(raSourceHandle, out DSP sourceDSP);
            jumpchannel.addDSP(FMOD.CHANNELCONTROL_DSP_INDEX.TAIL, sourceDSP);
            
            VECTOR pos = new VECTOR { x = 30, y = 0, z = 0 };
            VECTOR vel = new VECTOR { x = 0, y = 0, z = 0 };
            jumpchannel.set3DAttributes(ref pos, ref vel);
            jumpchannel.setPaused(false);

            




            // TODO: use this.Content to load your game content here
        }

        /// <summary>
        /// UnloadContent will be called once per game and is the place to unload
        /// game-specific content.
        /// </summary>
        protected override void UnloadContent()
        {
            // TODO: Unload any non ContentManager content here
            fmod.release();

        }

        /// <summary>
        /// Allows the game to run logic such as updating the world,
        /// checking for collisions, gathering input, and playing audio.
        /// </summary>
        /// <param name="gameTime">Provides a snapshot of timing values.</param>
        protected override void Update(GameTime gameTime)
        {
            if (GamePad.GetState(PlayerIndex.One).Buttons.Back == ButtonState.Pressed || Keyboard.GetState().IsKeyDown(Keys.Escape))
                Exit();

            // TODO: Add your update logic here
            fmod.update();
            base.Update(gameTime);
        }

        /// <summary>
        /// This is called when the game should draw itself.
        /// </summary>
        /// <param name="gameTime">Provides a snapshot of timing values.</param>
        protected override void Draw(GameTime gameTime)
        {
            GraphicsDevice.Clear(Color.CornflowerBlue);

            // TODO: Add your drawing code here

            base.Draw(gameTime);
        }
    }
}
 

Thanks again!

What version of FMOD and ResonanceAudio are you using?

One of the easier things you can do is check the FMOD_RESULT, which is returned by all the FMOD functions, to see if something is going wrong.

1 Like

Hello @cameron-fmod thanks for your answer.

I’m using the x86 version of the fdmod v2.0 preview posted on the downloads.
The resonance audio plugin, is the bundled with that version of Fmod api; I don’t changed this; only copied that, and like the other used dlls, is the x86 version.

I added to the code a small function to check the FMOD:RESULT object returned by the all fmod functions, but in the code all functions returned OK; any of that said something different or that one of them is wrong.

I uploaded the code to github if you want (and of course you can) check yourself what is wrong.

I Know that I can’t upload the fmod DLLS directly to a public repository, but I did that for have the exactly same environment that I have to reproduce the problem.

this is the repo:

Thanks for your help!

If you find something wrong please tell me!

I’ll check with a previews version (1.12.00) to see if I can make this works.

Thanks again!

To drive third party spatializer plugins in the Core API, set3DAttributes will not actually work.
It takes a significant amount of work to use the plugins this way, as you need to pass a specific struct (which would have to be found in the source code) in to the DSP using setDataParameter. This struct would include things like the sounds world position and the position relative to the listener among other things.

In general, we would normally recommend using the Studio API unless you have a specific requirement for the Core API.

In the Studio API, we handle all the information between the user and the third party plugins, allowing you to set the position on Event and we pass the required info to the DSP. The Studio API will also give you access to the FMOD Studio Authoring Tool.

Hello @cameron-fmod
My problem is that I’m blind, and the FMOD studio authoring application to create the banks, apply the plugins and something else isn’t compatible with the screen readers.
So because that, I’m finding a pure code solution to reproduce spatialiced 3d audio in my applications and FMOD low apis sounds very good to achieve that…
But if the plugins aren’t prepared to do that. Sadly, but the api itself works very nice.

So. I’m reading (on the fly) the source of the fmod connector of resonanse audio, and I can’t find the exactly parameter number (and struct) that determines the 3d properties of each sound source; the only I can find is properties for the gain, min and max distance and other things like that.

So really what you recommend to me?

Any of other spatialiced solutions that are compatible with FMOD like steam audio are implemented following and using the 3d properties of the FMOD itself?
Or at less have documentation of how to use the plugin with the fmod core api.
I don’t way the easy way; I only want to find a way to use this :slight_smile:
Thanks again for your answer!

PD: if you know where is that required struct on the source of resonanse, please tell me.

¡thanks!

All third party plugins are implemented the same way and won’t work with set3DAttributes.

I am currently going over how we set do this in the API. We have decided that we should add this information to our current docs, as it would be a good point of reference for people to use.

This will take some time as I go through the process of how to best implement this and provide all the information required (eg. a header for the ResonanceAudio plugin for information on the data structs).

Thank you for your patience and understanding.

1 Like

In the meantime, here is a basic idea of what you will need to do:

  • Create a FMOD_DSP_PARAMETER_3DATTRIBUTES, this stores an absolute and relative position.
  • You already have the absolute position from the set3DAttributes, you will just need to calculate the position relative to the listener.
  • Set this on the DSP by using DSP::setParameterData with the index of the kAttributes3D

This is the struct of parameters in order:

struct{
kGain = 0,
kSpread,
kDistanceMin,
kDistanceMax,
kDistanceModel,
kOcclusionIntensity,
kDirectivityAlpha,
kDirectivityOrder,
kAttributes3D,
kBypassRoom,
kEnableNearField,
kNearFieldGain,
kSourceOverallGain,
kNumSourceParameters
}

That should be all you need to do to get it working.

There are a number of other parameters there that you can use if you want.
You can find more of a description on them here:
https://resonance-audio.github.io/resonance-audio/develop/fmod/getting-started.html
Under the ‘Plugin features’ heading.

1 Like

Hello @cameron-fmod again!

Thanks really for your answers, and your effort to get that this things will be documented :slight_smile:
I really thanks you when that is done :slight_smile: so only when it occurs well. I have to stay tuned and checking the docs if a interesting update occurs xd.

So, about the answer:
Yep, I tried to do that, with fmod v2 and fmod v11012.
I can create the DSP_PARAMETER_3D_ATRIBUTE without problem, and I guess that I passed it correctly to the plugin using the setParameterData function, and well… convert the object from a struct instance to an array of bytes, I don’t know if I did that correctly.

But, I have a new problem; after set the parameter, the sounds is completely muted.

If I don’t set the parameter, I have a central mono sound, but after set the parameter the sound is muted. So…

What is wrong now?

This is the actual code

Again thanks for your answer!

It looks like you are setting the absolute and relative 3D Attributes to the same values before marshalling them.

Also, not a big deal but, you won’t need to call set3DAttributes on the Channel now as setParameterData will be handling the positional information.

hello @cameron-fmod
Oops, was my mistake, fixed it.
But, the problem wasn’t resolved.

Now, I’ve tried to put the sound source at 0, 0, 0, the same default location of the listener in the plugin.

So, if the source is at 0, 0, 0. and the listener at same position, the relativo position between them is 0, 0, 0 again.

Well, the expected result is have a centered mono sound at the center… but.
Noup, we have the same at the last experiment. Pure and noising silence.

The repo is updated again.

So, what is wrong now?

ps: I forgot to remove the calls to set3dAtributes, but… for the test, they don’t make any difference.

Thanks again for your feedback! And sorry for be anoying, but I wanna get this project working!

Thanks!

After a lot of digging and head scratching, it looks like the distance min and max values both default to zero, making the sound inaudible. As a test, using your code, I added this just after calling setParameterData:

errcheck(sourceDSP.setParameterFloat(2, 1.0f)); //Min distance
errcheck(sourceDSP.setParameterFloat(3, 100.0f)); //Max distance
1 Like

@cameron-fmod
How to send some love directly to you?

Man, it works. Very nice and without problem!

Thanks, thanks a lot for your help!

Now, I can play with the sound moving it to all places, or test with more sounds in the scene :slight_smile:

For the future will be a new fight pass to the listener room attributes, but. for now, it works very well.

So, probably will be interesting try to put this experience (with more details of course and other stuf) in a future documentation.

So again, thanks @cameron-fmod

For your help, and for your patience.

Bye!

1 Like