Routing Bink Video audio through FMod

Unreal Engine
1

Hi,
our game uses Fmod for all game audio and we are planning to use Bink to play video clips. In order to avoid relying upon the Unreal audio engine, we would like to route the video clips audio to FMod. I believe it should be possible to have Bink stream audio data into a buffer and then use System::createStream, but I am seeking help to address the details.
TIA,

2

Hi,

It is recommended to use Bink to play its audio alongside FMOD, is there a specific reason that you want to route the audio through FMOD?

3

Doing so we would need to leave the Unreal audio engine enabled in addition to Fmod and we would like to have only one engine. Consider that our project targets both PC and game consoles.

4

Hi,

That is understandable, it is possible to have UE audio and FMOD running side by side on console as well if that is a viable option? If not I can look into routing the Bink audio through FMOD.

Could I grab some info?
What version of FMOD and Unreal Engine are you using? Which consoles are you looking at packaging to?

5

We are using Unreal 5.3 (possibly upgrading to 5.4 during the development of the project) and FModStudio 2.02.18 (updgrade is also possible at this stage).

1 Like
6

Hi,

Thank you for the information. I got Bink and FMOD working together without any changes or issues on PS5. Were you looking at just targeting Sony or any other console?

7

I was told Bink would route the audio on Unreal Audio, and thus would require the Unreal Audio engine to be enabled, but indeed you’re right and Bink does not do that. We can in fact run Bink to play fullscreen video alongside Fmod even on consoles, without modifications and without enabling the Unreal Audio engine. However, I am still wondering if it’s possible to route Bink audio into an FMod event or bus. This could be used, for example, to play a video inside a game element such as a TV screen placed in the 3d scene and have FMod properly spatialize the audio track. We tried playing the audio track separately as a “regular” Fmod event, but audio and video cab easily get de-synced.

8

Hi,

Unfortunately, I cannot see a way to pass Bink audio to FMOD without significant code changes to both Bink and UE. I looked into implementing drift compensation on the side of Bink, but the API is limited and does not support changing the playback rate of the video.

An option may be using GetTime on the Bink media player and GetTimelinePosition on an FMOD Audio component and changing the event’s pitch to compensate for drift.

We have recently supported AudioLink which allows for UE sounds to be passed to the FMOD system: Unreal Integration: AudioLink. Let me know if this may be a viable option.

9

ouh, I also had an idea in this direction. Because I currently have 8 languages in which i have the audio for the same videos so my idea was to play the audio in fmod along with the video in unreal engine but i dont know wether this can be even in sync, ever. or is there a way? all languages audio will be deployed with the game so there will be no different versions of the game. But it doesn’t have to be bink in particular, just any video format that unreal supports, best out of the box.

10

Hi,

Yes, unfortunately, it’s more involved than I first thought. It requires accessing the FMOD Core API and influencing the FMOD::Chanel’s frequency rather than the event’s pitch.

let me know if this is still an option you would like to consider.

11

Uh, I know how to change the channel frequency, I do this already for my granular synth plugin. But what would be the logic to sync AV?

Best regards

12

Hi,

  1. Since we cannot get the samples directly from the Bink Media player we will need to infer how many/how far into the video we are. I found I was able to retrieve Total milliseconds passed:
    image
    from Bink. We can use this with Channel::getPosition to sync the event and video.
  2. Next from the event we will need to find the channel group with our channel
  3. Using the Video Playback example, we will set up the variables for both Bink and FMOD
UPROPERTY(BlueprintReadWrite)
float binkCurrentMiliseconds = 0;
float binkLastMiliseconds = 0;

unsigned int binkTotalSamplesWritten = 0;
unsigned int binkTotalSamplesRead = 0;
unsigned int binkMinimumSamplesWritten = 0xffffffff;
UPROPERTY(BlueprintReadWrite)
float playbackRate = 1.0;

unsigned int fmodTotalSamplesWritten = 0;
unsigned int fmodTotalSamplesRead = 0;
unsigned int fmodMinimumSamplesWritten = 0xffffffff;

unsigned int binkDriftThreshold = 0;
unsigned int binkTargetLatency = 0;
unsigned int binkAdjustedLatency = 0;
int binkActualLatency = 0;

unsigned int fmodDriftThreshold = 0;
unsigned int fmodTargetLatency = 0;
unsigned int fmodAdjustedLatency = 0;
int fmodActualLatency = 0;
  1. We will need to convert milliseconds to samples
unsigned int fmodMiliseconds = 0;
result = channel->getPosition(&fmodMiliseconds, FMOD_TIMEUNIT_MS);

binkLastMiliseconds = binkCurrentMiliseconds;

float binkSeconds = binkLastMiliseconds / 1000.0f;
float fmodSeconds = fmodMiliseconds / 1000.0f;

unsigned int binkSamplesWritten = binkSeconds * sampleRate;
binkTotalSamplesWritten += binkSamplesWritten;

unsigned int fmodSamplesWritten = fmodSeconds * sampleRate;
fmodTotalSamplesWritten += fmodSamplesWritten;
  1. We will then slightly change the Drift Compensation from the video playback for our situation.
if (binkSamplesWritten != 0 && (binkSamplesWritten < binkMinimumSamplesWritten))
{
    binkMinimumSamplesWritten = binkSamplesWritten;
    binkAdjustedLatency = fmax(binkSamplesWritten, binkTargetLatency);
}

if (fmodSamplesWritten != 0 && (binkSamplesWritten < fmodMinimumSamplesWritten))
{
    fmodMinimumSamplesWritten = binkSamplesWritten;
    fmodAdjustedLatency = fmax(fmodSamplesWritten, fmodTargetLatency);
}

int binkLatency = (int)binkTotalSamplesWritten - (int)binkTotalSamplesRead;
binkActualLatency = (int)((0.93f * binkActualLatency) + (0.03f * binkLatency));

int fmodLatency = (int)fmodTotalSamplesWritten - (int)fmodTotalSamplesRead;
fmodActualLatency = (int)((0.93f * fmodActualLatency) + (0.03f * fmodLatency));

int actualLatencyDifference = fmodActualLatency - binkActualLatency;
int frequency = sampleRate;
if (actualLatencyDifference < -50)
{
    frequency = sampleRate + (int)(sampleRate * (DRIFT_CORRECTION_PERCENTAGE / 100.0f));
    UE_LOG(LogTemp, Warning, TEXT("Slow down"));
}
else if (binkActualLatency > 50)
{
    frequency = sampleRate - (int)(sampleRate * (DRIFT_CORRECTION_PERCENTAGE / 100.0f));
    UE_LOG(LogTemp, Warning, TEXT("Speed Up"));
}
binkTotalSamplesRead += binkSamplesWritten;
fmodTotalSamplesRead += fmodSamplesWritten;
chan->setFrequency(frequency);

Hopefully, this will keep the Bink player and Event in sync.