I ran into a problem, in our game we use fmod to play sound, now there is a need to record a player’s battle and save it as a video. The plugins I used were ffmpeg and natcorder, but both of them had problems when dealing with the sound of fmod, and the sound of the recorded video was broken. All the data they need is the binary array of liner pcm, how can they get this data?
Hi,
There’s a couple of ways of doing this depending on what exactly you need:
FMOD_OUTPUTTYPE_WAVWRITERallows the FMOD System to write its mixer output to a wav file. Note that using this means that the system will not output to your regular audio output device, which may make it unsuitable for your purposes.- We have a Unity DSP Capture scripting example that shows how to extract the raw PCM data from a given DSP in FMOD’s signal chain, in this case the Master Channel Group. Unlike
WAVWRITER, this can be used even while outputting to audio devices.
I would recommend giving my reply in the following thread a read, as it broadly details how to do both things, including how to write the output of the DSP capture to wav if needed: Capturing Unity game’s FMOD output (5.1 / Busses) - #2 by Louis_FMOD
thanks for your replay.I tried the first method, saving it as a wav file and it successed,but fmod outputs it as a wav file when the RuntimeManager Destroyed, so I wrote a script to destroy the RuntimeManager when the screen recording stops.
public class RecFmodAudio : MonoBehaviour, IRecAudio
{
void Awake()
{
enabled = false;
}
public void StartRecording()
{
enabled = true;
startTime = Time.time;
Camera.main.gameObject.AddComponent<StudioListener>();
}
public void StopRecording(string savePath)
{
enabled = false;
var listner = Camera.main.gameObject.GetComponent<StudioListener>();
Destroy(listner);
var go = GameObject.Find("FMOD.UnityIntegration.RuntimeManager");
DestroyImmediate(go);
}
}
but I don’t want it to Destroy, I want to get the output of fmod in real time, is there any way? Looking at another example of ScriptUsageDspCapture you mentioned, how does CaptureDSPReadCallback get all of fmod’s output? Can you show the code? Thanks a million!!
ScriptUsageDspCapture creates a custom DSP which is placed on the FMOD System’s Master Channel Group, with a custom static read callback CaptureDSPReadCallback(). The read callback is called by the FMOD System with every mixer update and receives the audio signal that flows into the DSP, allowing you to directly access and/or modify the signal.
Additionally, you also pass the entire instance of the class ScriptUsageDspCapture to the DSP as user data. The class instance is then accessed from the user data within the callback, and the incoming audio signal is copied to the class member mDataBuffer. The data copied to mDataBuffer within the callback is then processed in ScriptUsageDspCapture.Update().
In the case of the scripting example, the data is used in Update() to draw the signal’s waveform, but for your purposes you’d instead want to accumulate all of the audio data and then presumably save it to file - that’s where the response linked in the other thread regarding writing to .wav comes in: record and output to .wav file - #2 by mathew
Yes, I have seen the c++ example, but what I didn’t understand is that in that example, I created a sound to record the sound of mirphone. What I need is to record the sound that has been recorded by the sound engineer and played by fmod. In this case, I don’t need to create a sound by myself, right? Therefore, I still do not understand how to record the output of fmod, I changed the sample code to this way, which is also not possible. So can you give me a code example?
//--------------------------------------------------------------------
//
// This is a Unity behaviour script that demonstrates how to capture
// the DSP with FMOD.DSP_READ_CALLBACK and how to create and add a DSP
// to master channel group ready for capturing.
//
// For the description of the channel counts. See
// https://fmod.com/docs/2.02/api/core-api-common.html#fmod_speakermode
//
// This document assumes familiarity with Unity scripting. See
// https://unity3d.com/learn/tutorials/topics/scripting for resources
// on learning Unity scripting.
//
//--------------------------------------------------------------------
using System;
using UnityEngine;
using System.Runtime.InteropServices;
using NatSuite;
using NatSuite.Recorders;
using NatSuite.Recorders.Clocks;
using FMODUnity;
using System.Collections.Generic;
using FMOD.Studio;
using System.Security.Cryptography;
public class ScriptUsageDspCapture : MonoBehaviour
{
private FMOD.DSP_READCALLBACK mReadCallback;
private FMOD.DSP mCaptureDSP;
private float mDataBuffer;
private GCHandle mObjHandle;
private uint mBufferLength;
private int mChannels = 0;
public MP4Recorder MP4Recorder;
public RealtimeClock RealtimeClock;
public bool StopRecord;
Dictionary<FMOD.ChannelGroup,FMOD.DSP> mDSPs;
[AOT.MonoPInvokeCallback(typeof(FMOD.DSP_READCALLBACK))]
static FMOD.RESULT CaptureDSPReadCallback(ref FMOD.DSP_STATE dsp_state, IntPtr inbuffer, IntPtr outbuffer, uint length, int inchannels, ref int outchannels)
{
FMOD.DSP_STATE_FUNCTIONS functions = (FMOD.DSP_STATE_FUNCTIONS)Marshal.PtrToStructure(dsp_state.functions, typeof(FMOD.DSP_STATE_FUNCTIONS));
IntPtr userData;
functions.getuserdata(ref dsp_state, out userData);
GCHandle objHandle = GCHandle.FromIntPtr(userData);
ScriptUsageDspCapture obj = objHandle.Target as ScriptUsageDspCapture;
// Save the channel count out for the update function
obj.mChannels = inchannels;
Debug.LogError($"channel {inchannels}");
// Copy the incoming buffer to process later
int lengthElements = (int)length * inchannels;
Marshal.Copy(inbuffer, obj.mDataBuffer, 0, lengthElements);
// Copy the inbuffer to the outbuffer so we can still hear it
Marshal.Copy(obj.mDataBuffer, 0, outbuffer, lengthElements);
return FMOD.RESULT.OK;
}
void Start()
{
mDSPs = new Dictionary<FMOD.ChannelGroup, FMOD.DSP>();
var res = RuntimeManager.StudioSystem.update();
var result = RuntimeManager.StudioSystem.flushCommands();
// Assign the callback to a member variable to avoid garbage collection
mReadCallback = CaptureDSPReadCallback;
// Allocate a data buffer large enough for 8 channels, pin the memory to avoid garbage collection
uint bufferLength;
int numBuffers;
FMODUnity.RuntimeManager.CoreSystem.getDSPBufferSize(out bufferLength, out numBuffers);
mDataBuffer = new float[bufferLength * 8];
mBufferLength = bufferLength;
// Get a handle to this object to pass into the callback
mObjHandle = GCHandle.Alloc(this);
if (mObjHandle != null)
{
// Define a basic DSP that receives a callback each mix to capture audio
FMOD.DSP_DESCRIPTION desc = new FMOD.DSP_DESCRIPTION();
desc.numinputbuffers = 1;
desc.numoutputbuffers = 1;
desc.read = mReadCallback;
desc.userdata = GCHandle.ToIntPtr(mObjHandle);
RuntimeManager.StudioSystem.getBankCount(out var numBanks);
RuntimeManager.StudioSystem.getBankList(out var banks);
for (int currentBank = 0; currentBank < numBanks; ++currentBank)
{
int numBusses = 0;
FMOD.Studio.Bus[] busses = null;
banks[currentBank].getBusCount(out numBusses);
banks[currentBank].getBusList(out busses);
for (int currentBus = 0; currentBus < numBusses; ++currentBus)
{
// Make sure the channel group of the current bus is assigned properly.
string busPath = null;
busses[currentBus].getPath(out busPath);
RuntimeManager.StudioSystem.getBus(busPath, out busses[currentBus]);
busses[currentBus].lockChannelGroup();
RuntimeManager.StudioSystem.flushCommands();
FMOD.ChannelGroup channelGroup;
busses[currentBus].getChannelGroup(out channelGroup);
RuntimeManager.CoreSystem.createDSP(ref desc, out var dsp);
mDSPs.Add(channelGroup, dsp);
channelGroup.addDSP(0, dsp);
busses[currentBus].unlockChannelGroup();
}
}
//// Create an instance of the capture DSP and attach it to the master channel group to capture all audio
//FMOD.ChannelGroup masterCG;
//if (FMODUnity.RuntimeManager.CoreSystem.getMasterChannelGroup(out masterCG) == FMOD.RESULT.OK)
//{
// if (FMODUnity.RuntimeManager.CoreSystem.createDSP(ref desc, out mCaptureDSP) == FMOD.RESULT.OK)
// {
// if (masterCG.addDSP(0, mCaptureDSP) != FMOD.RESULT.OK)
// {
// Debug.LogWarningFormat("FMOD: Unable to add mCaptureDSP to the master channel group");
// }
// }
// else
// {
// Debug.LogWarningFormat("FMOD: Unable to create a DSP: mCaptureDSP");
// }
//}
//else
//{
// Debug.LogWarningFormat("FMOD: Unable to create a master channel group: masterCG");
//}
}
else
{
Debug.LogWarningFormat("FMOD: Unable to create a GCHandle: mObjHandle");
}
}
void OnDestroy()
{
if (mObjHandle != null)
{
RuntimeManager.StudioSystem.getBankCount(out var numBanks);
RuntimeManager.StudioSystem.getBankList(out var banks);
for (int currentBank = 0; currentBank < numBanks; ++currentBank)
{
int numBusses = 0;
FMOD.Studio.Bus[] busses = null;
banks[currentBank].getBusCount(out numBusses);
banks[currentBank].getBusList(out busses);
for (int currentBus = 0; currentBus < numBusses; ++currentBus)
{
// Make sure the channel group of the current bus is assigned properly.
string busPath = null;
busses[currentBus].getPath(out busPath);
RuntimeManager.StudioSystem.getBus(busPath, out busses[currentBus]);
busses[currentBus].lockChannelGroup();
RuntimeManager.StudioSystem.flushCommands();
FMOD.ChannelGroup channelGroup;
busses[currentBus].getChannelGroup(out channelGroup);
if(mDSPs.ContainsKey(channelGroup))
channelGroup.removeDSP(mDSPs[channelGroup]);
busses[currentBus].unlockChannelGroup();
}
}
mObjHandle.Free();
}
}
const float WIDTH = 10.0f;
const float HEIGHT = 1.0f;
void Update()
{
// Do what you want with the captured data
if (mChannels != 0)
{
if (StopRecord == false)
MP4Recorder.CommitSamples(mDataBuffer, RealtimeClock.timestamp);
float yOffset = 5.7f;
for (int j = 0; j < mChannels; j++)
{
var pos = Vector3.zero;
pos.x = WIDTH * -0.5f;
for (int i = 0; i < mBufferLength; ++i)
{
pos.x += (WIDTH / mBufferLength);
pos.y = mDataBuffer[i + j * mBufferLength] * HEIGHT;
// Make sure Gizmos is enabled in the Unity Editor to show debug line draw for the captured channel data
Debug.DrawLine(new Vector3(pos.x, yOffset + pos.y, 0), new Vector3(pos.x, yOffset - pos.y, 0), Color.green);
}
yOffset -= 1.9f;
}
}
}
}
Different from the example code, I get all the banks here, and then get the bus inside each bank, take the channel group for each bus, and add dsp callback.
You don’t need to create the sound yourself in this case. The DSP Capture example gives you access to the audio data FMOD is currently playing. Instead of only using the current audio buffer, which is what the DSP Capture example does, you need to store all of audio data and then write it to file when playback has finished.
The example shows how to record from a microphone and write the microphone audio to a file. I linked it because specifically because it demonstrates how to write audio sample data to a WAV file - if you already have a way of writing your audio data to file, you can use that instead.
Thanks,I Choose the outputtype to WavWriter，then use ffmpeg to combine pics and voice to a video.
here, I have a problem, the video I recorded is only 1 minute 12 seconds, but the length of the output audio file is 2 minutes 0 seconds, why these two lengths are not the same? My approach is to set its output mode to WavWriter, and then destroy the RuntimeManager when stopping the screen recording.
So I want to know what factors affect the length of the output audio file, and how to make the length of the audio file and the length of the video file consistent, so that the sound and the picture are synchronized.
The only factor that should affect the length of the output audio file is how long the FMOD System exists after initialization. The most likely thing that would case a 48 second gap in lengths between the recorded audio and video is that the Unity game, and by extension the FMOD System, was playing for 48 seconds before you started video recording. You can confirm this by attempting to match your video to the relevant portions of the audio file - if they line up but there’s an excess of audio at the start of the file, then this is what has happened.
If this isn’t the case, and the audio file appears to be stretched in some way, can I get you to confirm your FMOD for Unity version number so I can try to reproduce the issue? Your previous posts indicate that you’re using 2.02.07, but just to be sure.
yes, version is 2.02.07,and unity version is 2021.3.19
I recorded another video. At the beginning, it could correspond with the picture, and then it got worse and worse. For a Time I output with Time. RealtimeSinceStartup. The initial value is 86 and the end value is 173, which should be 83 seconds, but the final audio output is 2 minutes and 31 seconds long.
I think of a possibility, is the time system of fmod consistent with the time system of unity? Could it be due to this inconsistency? The way I make videos is to take screenshots at regular intervals, save them as pictures, and then use ffmpeg to synthesize the video. The duration of the video is obtained by recording a start time using time.time and subtracting the start time from time.time when the recording stops.
Thanks for pointing out the issue - I’ve been able to reproduce it on my end and I’ve passed it along to the development team for further investigation. As a workaround, I would recommend either manually trimming the written wav file yourself, or using the DSP capture example I mentioned at the top of the thread (though you will need to handle writing to file yourself with in that case).
Thank you for your reply, but I don’t know how to capture the sound of all channels with DSP capture example, so I decided to remove the sound first.
If you mean FMOD Channel objects, then you just need to add the custom DSP on whatever Channels/ChannelGroups you want the audio from. An easy way to handle this is by using Buses, since you can retrieve their underlying ChannelGroups to place the DSP on with Studio::Bus::getChannelGroup.
If you mean the actual audio channels in the output signal, the audio signal is interleaved, meaning each “sample” contains one value for each channel in the audio signal. For more info, you can read over the Sample Data section of the FMOD Engine Glossary.
That said, if trimming the WAV yourself is adequate for your use-case, there’s no reason not to do that instead.