FMOD Pixel Streaming

Unreal Engine
1

Hi Guys,

I am new to the entire FMod system and wanted a question before I started exploring further. I’ve seen threads that try to capture FMod mix straight inside the pixel stream but this is not what I was looking for.

The pixel streaming module has direct support for pushing audio straight over a stream, essentially I just need a C++ interface to extract the raw audio data as its played in FMod. https://dev.epicgames.com/documentation/en-us/unreal-engine/API/Plugins/PixelStreaming/IPixelStreamingAudioInput

So if I can access the raw FMod data, I can send it directly. This is because inside my project, I have multiple streamed cameras.

2

So you are wanting to have multiple streamed cameras, and stream the audio data as heard from each of those camera positions to multiple clients simultaneously?

In FMOD, listeners are just points in space used for calculating the pan attributes of spatialized sounds. In the case of multiple listeners, the only way you can split the final mixed signal by listener is to zero out all but one listener with Studio::System::setListenerWeight.
You can then use a custom DSP with an FMOD_DSP_READ_CALLBACK to get the raw audio data from the master channel group, as you have read elsewhere, and send it along to the relevant client using IPixelStreamingAudioInput::PushAudio.

The problem is that this would make all other listeners silent, so you would only be able to do this for one listener at a time, which is not what you want if you are hoping to have multiple clients connecting to different camera positions simultaneously.
The way to work around this would be to have one FMOD::Studio::System object per camera, with each System playing its own copy of events. Keeping all of these events in sync however could be a challenge, and you are limited to a maximum of 8 System objects, so you could only have 8 cameras at once.

So it is possible to achieve a multi-camera multi-listener pixel streaming setup, but there are a lot of limitations and it would involve creating a lot of stuff that doesn’t come bundled with the FMOD Unreal integration. Please let me know if you have any further questions, or if you need any help getting any of this setup.

3

Hi Jeff,

That is actually really interesting to know! I think the limit will be the number of cameras first versus number of listeners you can have inside Unreal Engine, we see big FPS drops with 4 or more cameras.

The only question I would have with that is the performance impact of having multiple system objects inside FMOD, but your solution sounds exactly like what I want so thank you!

4

Each System won’t necessarily be playing all of the same events at all times due to the Virtual Voice System, but worst case scenario of n System playing all the same events would result in n times the memory and CPU of a single System.
One other thing I realized is that this would make use of the Unreal Sequencer more complicated, if you will be relying on that at all. You would need to make some integration changes to playback any sequenced FMOD Audio Components on multiple systems. Otherwise you could work around it by limiting yourself to a single camera / listener perspective during sequences.

1 Like
5

Is it possible to get some more documentation on creating systems and custom dsp’s? Ive been trying to get it setup today but couldnt manage

6 
class PixelDSP
{
private:
    FMOD::System* system;
    FMOD::DSP* dspInstance;
    TSharedPtr<IPixelStreamingAudioInput> input;
    static FMOD_RESULT F_CALLBACK DSPCallback(FMOD_DSP_STATE* dsp_state, float* inbuffer, float* outbuffer, unsigned int length, int inchannels, int* outchannels);

public:
    PixelDSP(FMOD::System* system, TSharedPtr<IPixelStreamingAudioInput> input);
    void UpdateListenerPosition(FMOD_VECTOR position, FMOD_VECTOR velocity, FMOD_VECTOR forward, FMOD_VECTOR up);
    static PixelDSP* GetInstance(FMOD_DSP_STATE* dsp_state);
};

// Implementation

FMOD_RESULT F_CALLBACK PixelDSP::DSPCallback(FMOD_DSP_STATE* dsp_state, float* inbuffer, float* outbuffer, unsigned int length, int inchannels, int* outchannels)
{
    PixelDSP* instance = GetInstance(dsp_state);
    if (instance)
    {
        UE_LOG(LogTemp, Warning, TEXT("Length: %d"), length);
        UE_LOG(LogTemp, Warning, TEXT("Inchannels: %d"), inchannels);
        UE_LOG(LogTemp, Warning, TEXT("Outchannels: %d"), *outchannels);
        // Process audio data here
        memcpy(outbuffer, inbuffer, length * inchannels * sizeof(float));
        // Push audio data to pixel stream here
        if (instance->input.IsValid())
        {
            UE_LOG(LogTemp, Warning, TEXT("Pushing audio to pixel stream"));
            instance->input->PushAudio(outbuffer, length * inchannels, inchannels, 48000);
        }
    }
    return FMOD_OK;
}

PixelDSP::PixelDSP(FMOD::System* system, TSharedPtr<IPixelStreamingAudioInput> input) : system(system), dspInstance(nullptr), input(input)
{
    FMOD_DSP_DESCRIPTION dspDesc;
    memset(&dspDesc, 0, sizeof(FMOD_DSP_DESCRIPTION));
    strcpy(dspDesc.name, "PixelDSP");
    dspDesc.version = 0x00010000;
    dspDesc.numinputbuffers = 1;
    dspDesc.numoutputbuffers = 1;
    dspDesc.read = PixelDSP::DSPCallback;
    dspDesc.userdata = this;

    system->createDSP(&dspDesc, &dspInstance);
    system->set3DNumListeners(1);
}

PixelDSP* PixelDSP::GetInstance(FMOD_DSP_STATE* dsp_state)
{
    return static_cast<PixelDSP*>(dsp_state->plugindata);
}

void PixelDSP::UpdateListenerPosition(FMOD_VECTOR position, FMOD_VECTOR velocity, FMOD_VECTOR forward, FMOD_VECTOR up)
{
    system->set3DListenerAttributes(0, &position, &velocity, &forward, &up);
}

This is my current setup to essentially have a listener inside a system and get the output data from it. Currently I dont get any data and the log statements aren’t called.

The way im playing my audio is an audio component inside my actor

7

Thank you for sharing the code snippet, I can see a few things that would be preventing your DSP from working correctly.

The FMOD_DSP_STATE::plugindata is different to the FMOD_DSP_DESCRIPTION::userdata you passed into the DSP description. To access the userdata you can call the getuserdata DSP state function:

FMOD_RESULT F_CALLBACK PixelDSP::DSPCallback(FMOD_DSP_STATE* dsp_state, float* inbuffer, float* outbuffer, unsigned int length, int inchannels, int* outchannels)
{
+   PixelDSP* instance;
+   dsp_state->functions->getuserdata(dsp_state, (void**)&instance);
    if (instance)
    {
...

Also, I see you are creating the DSP, but you also need to connect the DSP instance to something for it to start receiving audio. To do this, you first need to find something to attach it to- System::getMasterChannelGroup will give you a ChannelGroup that has all audio routed into it, which should be suitable in your case. Then you need to add the DSP to the ChannelGroup, which you can do using ChannelGroup:addDSP. Here are the lines you can add to the PixelDSP constructor to get it working:

PixelDSP::PixelDSP(FMOD::System* system, TSharedPtr<IPixelStreamingAudioInput> input) : system(system), dspInstance(nullptr), input(input)
{
...
    system->createDSP(&dspDesc, &dspInstance);

    FMOD::ChannelGroup* masterGroup;
+   system->getMasterChannelGroup(&masterGroup);
+   mastergroup->addDSP(0, dspInstance);

    system->set3DNumListeners(1);
}

Can you please give those changes a shot and let me know if the DSP still isn’t working?

1 Like
8

As for documentation, there isn’t much explaining how to create DSPs admittedly, but we do have some C++ Core API examples which you can download from the FMOD Engine section of our Downloads page.
After installation they can be found in “C:\Program Files (x86)\FMOD SoundSystem\FMOD Studio API Windows\api\core\examples\vs2019”. I would recommend you take a look at the dsp_custom and fmod_gain examples for a reference on what all the callbacks are for and what you can do with them.

1 Like
9 
#include "FMODStudioModule.h"
#include "IPixelStreamingStreamer.h"
#include "IPixelStreamingAudioInput.h"
#include "fmod_studio.hpp"
#include "fmod.hpp"
#include "fmod_errors.h"
#include <cstring>
 
class PixelDSP
{
private:
    FMOD::Studio::System* studioSystem;
    FMOD::System* system;
    FMOD::DSP* dspInstance;
    FMOD::Studio::Bank* bank;
    FMOD::Studio::EventInstance* event;
    TSharedPtr<IPixelStreamingAudioInput> input;
 
public:
    PixelDSP(FMOD::Studio::System* studioSystem, TSharedPtr<IPixelStreamingAudioInput> input);
    static FMOD_RESULT DSPCallback(FMOD_DSP_STATE* dsp_state, float* inbuffer, float* outbuffer, unsigned int length, int inchannels, int* outchannels);
    void UpdateListenerPosition(FMOD_VECTOR position, FMOD_VECTOR velocity, FMOD_VECTOR forward, FMOD_VECTOR up);
    static PixelDSP* GetInstance(FMOD_DSP_STATE* dsp_state);
    void LoadBank(const char* relativeBankPath);
    void LoadEvent(const char* eventPath);
    void PlayEvent();
    void UpdateEventLocation(FMOD_VECTOR position, FMOD_VECTOR velocity, FMOD_VECTOR forward, FMOD_VECTOR up);
    void UpdateEventParameter(const char* parameterName, float value);
    void Update();
 
};
 
// Implementation
 
FMOD_RESULT F_CALLBACK PixelDSP::DSPCallback(FMOD_DSP_STATE* dsp_state, float* inbuffer, float* outbuffer, unsigned int length, int inchannels, int* outchannels)
{
 
    PixelDSP* instance = GetInstance(dsp_state);
    if (instance)
    {
 
        // Push audio data to pixel stream here
        if (instance->input.IsValid())
        {
            for (int i = 0; i < length * inchannels; i++)
            {
 
                if (outbuffer[i] != 0) {
                    UE_LOG(LogTemp, Warning, TEXT("Audio: %f"), outbuffer[i]);
                }
            }
            instance->input->PushAudio(outbuffer, length * inchannels, inchannels, 48000);
        }
    }
    return FMOD_OK;
}
 
PixelDSP::PixelDSP(FMOD::Studio::System* studioSystem, TSharedPtr<IPixelStreamingAudioInput> input) : studioSystem(studioSystem), system(nullptr), dspInstance(nullptr), input(input)
{
    FMOD_DSP_DESCRIPTION dspDesc;
    memset(&dspDesc, 0, sizeof(FMOD_DSP_DESCRIPTION));
    strcpy(dspDesc.name, "PixelDSP");
    dspDesc.version = 0x00010000;
    dspDesc.numinputbuffers = 1;
    dspDesc.numoutputbuffers = 1;
 
    dspDesc.read = PixelDSP::DSPCallback;
    dspDesc.userdata = this;
 
    studioSystem->initialize(32, FMOD_STUDIO_INIT_LIVEUPDATE, FMOD_INIT_NORMAL, nullptr);
    studioSystem->getCoreSystem(&system);
 
    FMOD_RESULT result;
    system->createDSP(&dspDesc, &dspInstance);
    // Add the DSP to the master channel group to ensure it's processed
    FMOD::ChannelGroup* masterGroup = nullptr;
    result = system->getMasterChannelGroup(&masterGroup);
    if (result != FMOD_OK)
    {
        UE_LOG(LogTemp, Error, TEXT("Failed to get master channel group: %s"), FMOD_ErrorString(result));
        return;
    }
 
    result = masterGroup->addDSP(0, dspInstance);
    if (result != FMOD_OK)
    {
        UE_LOG(LogTemp, Error, TEXT("Failed to add DSP to master group: %s"), FMOD_ErrorString(result));
    }
 
    // Set the number of 3D listeners
    result = system->set3DNumListeners(1);
    if (result != FMOD_OK)
    {
        UE_LOG(LogTemp, Error, TEXT("Failed to set number of listeners: %s"), FMOD_ErrorString(result));
    }
 
    UE_LOG(LogTemp, Warning, TEXT("DSP initialized successfully"));
}
 
PixelDSP* PixelDSP::GetInstance(FMOD_DSP_STATE* dsp_state)
{
    FMOD::DSP* dsp;
    dsp = static_cast<FMOD::DSP*>(dsp_state->instance);
    if (dsp)
    {
        void *userData;
        dsp->getUserData(&userData);
        return static_cast<PixelDSP*>(userData);
    }
    UE_LOG(LogTemp, Error, TEXT("Failed to get DSP instance"));
    return nullptr;
}
 
void PixelDSP::UpdateListenerPosition(FMOD_VECTOR position, FMOD_VECTOR velocity, FMOD_VECTOR forward, FMOD_VECTOR up)
{
    system->set3DListenerAttributes(0, &position, &velocity, &forward, &up);
}
 
void PixelDSP::LoadBank(const char* filename)
{
    FMOD_RESULT result = studioSystem->loadBankFile(filename, FMOD_STUDIO_LOAD_BANK_NORMAL, &bank);
    if (result != FMOD_OK)
    {
        FString WideFilename = FString(UTF8_TO_TCHAR(FMOD_ErrorString(result)));
        UE_LOG(LogTemp, Error, TEXT("Failed to load bank: %s"), *WideFilename);
    }
    result = bank->loadSampleData();
    if (result != FMOD_OK)
    {
        FString ErrorString = FString(UTF8_TO_TCHAR(FMOD_ErrorString(result)));
        UE_LOG(LogTemp, Error, TEXT("Failed to load sample data: %s"), *ErrorString);
    }
}
 
void PixelDSP::LoadEvent(const char* eventPath)
{
    FMOD::Studio::EventDescription* eventDescription;
    FMOD_RESULT result = studioSystem->getEvent(eventPath, &eventDescription);
    if (result != FMOD_OK)
    {
        FString WideFilename = FString(UTF8_TO_TCHAR(FMOD_ErrorString(result)));
        UE_LOG(LogTemp, Error, TEXT("Failed to get event: %s"), *WideFilename);
    }
    eventDescription->createInstance(&event);
}
 
void PixelDSP::PlayEvent()
{
    FMOD_RESULT result = event->start();
    if (result != FMOD_OK)
    {
        FString WideFilename = FString(UTF8_TO_TCHAR(FMOD_ErrorString(result)));
        UE_LOG(LogTemp, Error, TEXT("Failed to start event: %s"), *WideFilename);
    }
}
 
void PixelDSP::UpdateEventLocation(FMOD_VECTOR position, FMOD_VECTOR velocity, FMOD_VECTOR forward, FMOD_VECTOR up)
{
    FMOD_3D_ATTRIBUTES attributes;
    attributes.position = position;
    attributes.velocity = velocity;
    attributes.forward = forward;
    attributes.up = up;
    event->set3DAttributes(&attributes);
}
 
void PixelDSP::UpdateEventParameter(const char* parameterName, float value)
{
    FMOD_RESULT result = event->setParameterByName(parameterName, value);
    if (result != FMOD_OK)
    {
        FString WideFilename = FString(UTF8_TO_TCHAR(FMOD_ErrorString(result)));
        UE_LOG(LogTemp, Error, TEXT("Failed to set parameter: %s"), *WideFilename);
    }
}
 
void PixelDSP::Update()
{
    studioSystem->update();
}

Thanks, that seemed to get me some more progress! The part that I struggle with now is it complains about buses and I never get any audio data?

10

I see you removed the line:

memcpy(outbuffer, inbuffer, length * inchannels * sizeof(float));

So nothing is populating the output buffer, which you are then pushing to IPixelStreamingAudioInput. I think adding that memcpy line back in should give you audio data, or pushing/logging the input buffer in the read callback instead.

if (instance->input.IsValid())
{
    for (int i = 0; i < length * inchannels; i++)
    {
+       if (inbuffer[i] != 0) {
+           UE_LOG(LogTemp, Warning, TEXT("Audio: %f"), inbuffer[i]);
        }
+   }    instance->input->PushAudio(inbuffer, length * inchannels, inchannels, 48000);
}

It’s also no clear to me whether any FMOD::Sounds or FMOD::Studio::EventInstances are playing on this System. Can you please confirm that this system, and not some other one, is being used to play back sounds or events?
As for the bus problem, which line is getting the error, and what is the error code you are getting?

1 Like
11

Hi Jeff,

Just wanted to say thanks for all your help, I’ve now managed to get multiple cameras streaming audio!

It would be nice to get other audio from other cameras at the same time as right now I have to play the sound direct to the specific system but for now this is a massive start from what I had before!

Once again, thanks for all your support and amazing stuff you guys do! Have a wonderful week

12

Great to hear you’ve got it working!

If you also want to get the signal from any other position you can use Multiple Listeners with each of your system objects. Just keep in mind that you won’t be able to get the unique signal of each listener at the same time (which is why I suggested the multiple system approach), though you can get one at a time using Listener Weights. Our “3d_multi” Studio API example demonstrates how to use multiple listeners, and can be found in the “api\studio\examples\vs2019” directory of your FMOD Engine installation.

Please let us know if you run into any other issues, and have a wonderful week yourself!

13

Hi Jeff,

What I mean is that if I have one object that I am playing a sound to, I would also have to play this sound to the other 7 Systems at the same time to make sure the listeners in the different systems also heard the sound.

As far as I am aware, there is no way to sync the systems so that if I play a sound in one system, the other systems are aware of it?

14

Thanks for the clarification, I see what you mean now.
Yes this is a limitation of the multi-system approach, and we don’t have a way to sync multiple systems unfortunately.