Can't record with Unity Recorder

Unity
1

I want to record game footage and audio with the Unity recorder, but now that I’m using FMOD, I can’t anymore. How can I record my game in Unity? The Unity Recorder did work well because it slowed the timesteps to preserve 60 fps, producing a great, smooth video.

I will add that recording with OBS (a) does not do this, producing inconsistent framerates that make the video far worse and (b) doesn’t have usable audio as we’re developing on MacOS and OBS can’t access system audio.

If this isn’t solvable, my team will have to dump FMOD. The ability to share footage of the game is critical to talking about it and promoting it.

2

Unfortunately the Unity Recorder doesn’t hook into FMOD audio, and that’s not really something we have control over. You should be able to hook in system audio from Unity into OBS using Loopback, but if OBS isn’t suiting your needs in terms of video quality there’s not much else I can suggest.

3

that this isn’t working is such a shame :expressionless: any updates in 2024?

4

It is possible to pass audio data to Unity through an OnAudioFilterRead callback.
You can get audio out of FMOD in various ways, I think the simplest would be to implement a custom DSP, implementing a FMOD_DSP_READ_CALLBACK that copies all of FMOD’s audio data into an intermediate buffer. You could then copy that buffer in the OnAudioFilterRead callback, and get FMOD outputting to Unity, and thus through the unity Recorder.

Here is a basic example:

ScriptUsageTap.cs 
//--------------------------------------------------------------------
// Basic FMOD to Unity Audio example.
// This script copies audio data from FMOD's master channel group to
// Unity's audio buffer using OnAudioFilterRead.
// Prerequisties:
// 1. Unity audio is enabled.
// 2. The GameObject also contains a Unity AudioSource.
// 3. There is a Unity AudioListener on a different GameObject.
// 4. Your master bus is built with a Stereo channel layout.
// Use of this script will incur one mix block of latency.
//--------------------------------------------------------------------

using UnityEngine;
using System.Runtime.InteropServices;
using System.Collections.Generic;
using System;

[RequireComponent(typeof(AudioSource))]
public class ScriptUsageTap : MonoBehaviour
{
    private FMOD.DSP_DESCRIPTION mDSPDesc;
    private FMOD.DSP mDSP;

    private List<float> mBuffer = new List<float>();
    private bool mRunning = false;

    private void CHECK_RESULT(FMOD.RESULT result, [System.Runtime.CompilerServices.CallerLineNumber] int sourceLineNumber = 0)
    {
        if (result != FMOD.RESULT.OK)
        {
            Debug.LogError(String.Format("Call to FMOD API failed with result: \"{0}\" line: {1}", FMOD.Error.String(result), sourceLineNumber));
        }
    }

    void OnAudioFilterRead(float[] data, int channels)
    {
        if (!mRunning)
            return;

        if (mBuffer.Count >= data.Length)
        {
            // Copy from intermediate buffer into Unity's audio buffer
            Array.Copy(mBuffer.GetRange(0, data.Length).ToArray(), data, data.Length);
            mBuffer.RemoveRange(0, data.Length);
        }
    }

    private void Start()
    {
        mDSPDesc = new FMOD.DSP_DESCRIPTION();
        mDSPDesc.numinputbuffers = 1;
        mDSPDesc.numoutputbuffers = 1;
        mDSPDesc.read = (ref FMOD.DSP_STATE dsp_state, IntPtr inbuffer, IntPtr outbuffer, uint length, int inchannels, ref int outchannels) =>
        {
            float[] tmpBuffer = new float[length * inchannels];

            if (length > 0)
            {
                // Copy to intermediate buffer
                Marshal.Copy(inbuffer, tmpBuffer, 0, tmpBuffer.Length);
                mBuffer.AddRange(tmpBuffer);

                // Silence FMOD output
                Array.Clear(tmpBuffer, 0, tmpBuffer.Length);
                Marshal.Copy(tmpBuffer, 0, outbuffer, tmpBuffer.Length);
            }

            return FMOD.RESULT.OK;
        };

        CHECK_RESULT(FMODUnity.RuntimeManager.CoreSystem.createDSP(ref mDSPDesc, out mDSP));
    }

    private void OnDestroy()
    {
        CHECK_RESULT(FMODUnity.RuntimeManager.CoreSystem.getMasterChannelGroup(out FMOD.ChannelGroup group));
        CHECK_RESULT(group.removeDSP(mDSP));
        CHECK_RESULT(mDSP.release());
        mRunning = false;
    }

    private void Update()
    {
        FMODUnity.RuntimeManager.CoreSystem.getMasterChannelGroup(out FMOD.ChannelGroup group);
        if (group.hasHandle() && !mRunning)
        {
            mRunning = true;
            CHECK_RESULT(group.addDSP(0, mDSP));
        }
    }
}

if you want to support more channel layouts you will have to implement some custom downmixing behavior yourself.

5

Maybe you can use one of the suggested external softwares that are mentioned by Unity.

The snipper tool for windows also records the system audio and depending on your audio configuration from window you may be able to record surround sound and not just Stereo.
For surround sound I don’t think you’d need an audio interface with 6 or more outputs as long as you can configure it from the audio configuration panel from windows.
Also you could use Voicemeeter VB-Audio VoiceMeeter
This software will act as a virtual mixer with enough outputs for you to monitor your audio in stereo or surround. You can set up a loop for your audio from FMOD output to voicemmeeter back into the input of snippertool.

6

Here is script that I use for my proj. I was stumbling into same issue and had to develop custom solution to be able use Unity Recorder and record and record Unity Audio.

Attach Camera Recorder to camera which you wanna use to be recorded. Note that it must NOT be you gameplay camera, create a new one and place it where you want.

System requires Unity Recorder to be installed. Also it requires FFMPEG to be intsalled and you will need to set path to it in the inspector. FFMPEG is used to combine recorded result.

Component separately records Video and Audio from FMOD, and then combines them into one file. You can choose whether to delete or keep original files in case you want to compose it manual.

Files are stored in Proj/Recordings

Note: Since Unity Recorder is Editor only tool, this approach wont work in build. You will also need to enable Unity Audio for propper use of Unity Recorder (typicaly disabled with FMOD)

After recording is done script waits for amount of seconds set in “waitBeforeTryToMerge” before perform merge. This is due no callback from Unity Recorder once file saved and stored on disc.

CameraRecorder classs

using System;
using UnityEngine;
using UnityEngine.InputSystem;
using System.Collections;
using System.Diagnostics;
using System.IO;
using UnityEditor.Recorder;
using UnityEditor.Recorder.Encoder;
using UnityEditor.Recorder.Input;
using Debug = UnityEngine.Debug;

namespace Recorder
{
    [RequireComponent(typeof(Camera))] // Ensure a Camera component is attached
    public class CameraRecorder : MonoBehaviour
    {
        [SerializeField] private int width = 1920;
        [SerializeField] private int height = 1080;
        [SerializeField] private float frameRate = 60; // Set target FPS to 60
        [SerializeField] private string outputFileName = "RecordedVideo";
        [SerializeField] private InputActionReference recordingAction;
        [SerializeField] private bool recordAudio = true;
        [SerializeField] private bool deleteSourceFiles = false;
        [SerializeField] private float waitBeforeTryToMerge = 2.0f;
        [SerializeField] private string pathToFFMPEG = "/opt/homebrew/bin/ffmpeg";

        private Camera cameraComponent;
        private RenderTexture temporaryRenderTexture;
        private RecorderController recorderController;
        private AudioRecorder audioRecorder;
        private bool isRecording = false;
        private string outputDirectory;
        private string videoFilePath;
        private string audioFilePath;

        private void Start()
        {
            cameraComponent = GetComponent<Camera>();

            if (recordAudio)
            {
                audioRecorder = new AudioRecorder();
            }

            recordingAction.action.Enable();
            recordingAction.action.performed += context =>
            {
                if (isRecording)
                {
                    StopRecording();
                }
                else
                {
                    StartRecording();
                }
            };
        }

        private void SetupRecorder()
        {
            var fullName = Directory.GetParent(Application.dataPath)?.FullName;
            if (fullName != null)
                outputDirectory = Path.Combine(fullName, "Recordings");
            Directory.CreateDirectory(outputDirectory);

            var dateTimeSuffix = DateTime.Now.ToString("MM.dd.HH.mm.ss");
            var outputFileCombined = $"{outputFileName}_{dateTimeSuffix}";
            
            videoFilePath = Path.Combine(outputDirectory, outputFileCombined);
            audioFilePath = Path.ChangeExtension(videoFilePath, ".wav");

            // Initialize the temporary RenderTexture
            temporaryRenderTexture = new RenderTexture(width, height, 24);
            temporaryRenderTexture.format = RenderTextureFormat.ARGB32;
            cameraComponent.targetTexture = temporaryRenderTexture;

            var renderTextureInput = new RenderTextureInputSettings
            {
                RenderTexture = temporaryRenderTexture
            };

            var videoRecorderSettings = ScriptableObject.CreateInstance<MovieRecorderSettings>();
            videoRecorderSettings.name = "CustomVideoRecorder";
            videoRecorderSettings.Enabled = true;
            videoRecorderSettings.EncoderSettings = new CoreEncoderSettings();
            videoRecorderSettings.ImageInputSettings = renderTextureInput;
            videoRecorderSettings.OutputFile = videoFilePath;
            videoRecorderSettings.FrameRatePlayback = FrameRatePlayback.Constant;
            videoRecorderSettings.FrameRate = frameRate;
            videoRecorderSettings.CapFrameRate = true;

            var recorderControllerSettings = ScriptableObject.CreateInstance<RecorderControllerSettings>();
            recorderControllerSettings.AddRecorderSettings(videoRecorderSettings);
            recorderControllerSettings.SetRecordModeToManual();
            recorderControllerSettings.FrameRate = frameRate;
            recorderControllerSettings.CapFrameRate = true;

            recorderController = new RecorderController(recorderControllerSettings);
        }

        private void StartRecording()
        {
            SetupRecorder();
            
            isRecording = true;

            // Ensure the temporary RenderTexture is available
            if (temporaryRenderTexture == null)
            {
                temporaryRenderTexture = new RenderTexture(width, height, 24);
                cameraComponent.targetTexture = temporaryRenderTexture;
            }

            recorderController.PrepareRecording();
            recorderController.StartRecording();

            if (recordAudio)
            {
                audioRecorder.StartRecording();
            }
        }

        private void StopRecording()
        {
            isRecording = false;
            if (recordAudio)
            {
                audioRecorder.StopRecording();
            }
            recorderController.StopRecording();

            StartCoroutine(SaveAudio());

            // Clean up the temporary RenderTexture
            if (temporaryRenderTexture != null)
            {
                cameraComponent.targetTexture = null;
                Destroy(temporaryRenderTexture);
                temporaryRenderTexture = null;
            }

            if (!recordAudio) return;
            StartCoroutine(WaitForRecordingToFinish());
        }
        
        private IEnumerator WaitForRecordingToFinish()
        {
            yield return new WaitForSeconds(waitBeforeTryToMerge);

            //We do not add extension too path initially due Recorder structure.
            //It does it inside Recorder. But we need that after recording is done.
            videoFilePath += ".mp4";
            
            if (File.Exists(videoFilePath))
            {
                Debug.Log("Recording finished. Starting to combine video and audio.");
                CombineVideoAndAudio();
            }
            else
            {
                Debug.LogError("Recorded video file was not found after it was recorded.");
            }
        }

        private IEnumerator SaveAudio()
        {
            if (recordAudio)
            {
                audioRecorder.SaveAudioToWav(audioFilePath);
                yield return null;

                Debug.Log("Audio saved to " + audioFilePath);
            }

            Debug.Log("Recording completed.");
        }

        private void CombineVideoAndAudio()
        {
            var combinedFilePath = Path.Combine(Path.GetDirectoryName(videoFilePath),
                Path.GetFileNameWithoutExtension(videoFilePath) + "_Combined" + Path.GetExtension(videoFilePath)
            );
            
            // Build the FFmpeg command
            var ffmpegCommand = $"-i \"{videoFilePath}\" -i \"{audioFilePath}\" -c:v copy -c:a aac -strict experimental \"{combinedFilePath}\"";

            var process = new Process
            {
                StartInfo = new ProcessStartInfo
                {
                    FileName = pathToFFMPEG,  // Ensure ffmpeg is accessible from PATH
                    Arguments = ffmpegCommand,
                    RedirectStandardOutput = true,
                    RedirectStandardError = true,
                    UseShellExecute = false,
                    CreateNoWindow = true
                }
            };

            try
            {
                process.Start();

                string output = process.StandardOutput.ReadToEnd();
                string error = process.StandardError.ReadToEnd();

                process.WaitForExit();

                if (process.ExitCode == 0)
                {
                    Debug.Log("Combined video and audio saved to " + combinedFilePath);
                    if (deleteSourceFiles)
                    {
                        DeleteSourceFiles();
                    }
                }
                else
                {
                    Debug.LogError("FFmpeg failed with exit code " + process.ExitCode);
                    Debug.LogError("FFmpeg Output: " + output);
                    Debug.LogError("FFmpeg Errors: " + error);
                }
            }
            catch (Exception ex)
            {
                Debug.LogError("An error occurred while combining video and audio: " + ex.Message);
            }
        }

        private void DeleteSourceFiles()
        {
            try
            {
                if (File.Exists(videoFilePath))
                {
                    File.Delete(videoFilePath);
                    Debug.Log("Deleted video file: " + videoFilePath);
                }

                if (File.Exists(audioFilePath))
                {
                    File.Delete(audioFilePath);
                    Debug.Log("Deleted audio file: " + audioFilePath);
                }
            }
            catch (Exception ex)
            {
                Debug.LogError("An error occurred while deleting source files: " + ex.Message);
            }
        }
    }
}

AudioRecorder class

using System;
using System.Collections.Generic;
using System.IO;
using System.Runtime.InteropServices;
using FMODUnity;

namespace Recorder
{
    public class AudioRecorder
    {
        private FMOD.DSP mDSP;
        private FMOD.ChannelGroup mCg;
        private readonly List<float> mAudioData;
        private readonly int mSampleRate;
        private int mNumChannels;
        private FMOD.DSP_DESCRIPTION mDSPDescription;

        public AudioRecorder()
        {
            mAudioData = new List<float>();
            RuntimeManager.CoreSystem.getSoftwareFormat(out mSampleRate, out _, out _);

            var mObjHandle = GCHandle.Alloc(this, GCHandleType.Pinned);
            mDSPDescription = new FMOD.DSP_DESCRIPTION
            {
                numinputbuffers = 1,
                numoutputbuffers = 1,
                read = CaptureDSPReadCallback,
                userdata = GCHandle.ToIntPtr(mObjHandle)
            };
        }

        public void StartRecording()
        {
            mAudioData.Clear();

            var bus = RuntimeManager.GetBus("bus:/");
            if (bus.getChannelGroup(out mCg) != FMOD.RESULT.OK) return;
            RuntimeManager.CoreSystem.createDSP(ref mDSPDescription, out mDSP);
            mCg.addDSP(0, mDSP);
        }

        public void StopRecording()
        {
            if (!mDSP.hasHandle()) return;
            mCg.removeDSP(mDSP);
            mDSP.release();
        }

        public void SaveAudioToWav(string filePath)
        {
            using var fs = File.Create(filePath);
            using var bw = new BinaryWriter(fs);
            WriteWavHeader(bw, mAudioData.Count);
            var bytes = new byte[mAudioData.Count * 4];
            Buffer.BlockCopy(mAudioData.ToArray(), 0, bytes, 0, bytes.Length);
            fs.Write(bytes, 0, bytes.Length);
        }

        [AOT.MonoPInvokeCallback(typeof(FMOD.DSP_READ_CALLBACK))]
        private static FMOD.RESULT CaptureDSPReadCallback(ref FMOD.DSP_STATE dspState, IntPtr inBuffer, IntPtr outBuffer, uint length, int inChannels, ref int outChannels)
        {
            var lengthElements = (int)length * inChannels;
            var data = new float[lengthElements];
            Marshal.Copy(inBuffer, data, 0, lengthElements);

            var functions = (FMOD.DSP_STATE_FUNCTIONS)Marshal.PtrToStructure(dspState.functions, typeof(FMOD.DSP_STATE_FUNCTIONS));
            functions.getuserdata(ref dspState, out var userData);

            if (userData != IntPtr.Zero)
            {
                var objHandle = GCHandle.FromIntPtr(userData);

                if (objHandle.Target is AudioRecorder { mAudioData: { } } obj)
                {
                    obj.mNumChannels = inChannels;
                    obj.mAudioData.AddRange(data);
                }
            }

            Marshal.Copy(data, 0, outBuffer, lengthElements);

            return FMOD.RESULT.OK;
        }

        private void WriteWavHeader(BinaryWriter bw, int length)
        {
            bw.Seek(0, SeekOrigin.Begin);

            bw.Write(System.Text.Encoding.ASCII.GetBytes("RIFF"));
            bw.Write(32 + length * 4 - 8);
            bw.Write(System.Text.Encoding.ASCII.GetBytes("WAVEfmt "));
            bw.Write(16);
            bw.Write((short)3);
            bw.Write((short)mNumChannels);
            bw.Write(mSampleRate);
            bw.Write(mSampleRate * 32 / 8 * mNumChannels);
            bw.Write((short)(32 / 8 * mNumChannels));
            bw.Write((short)32);
            bw.Write(System.Text.Encoding.ASCII.GetBytes("data"));
            bw.Write(length * 4);
        }
    }
}
7

This solution on paper seems the most feasible to keep using the Unity Recorder package and having FMOD audio but tried this script (Unity v2022.3) and when active in play mode I just hear some high pitch clicks and nothing more. Destroying the object reverts the audio to normal.

I think I did everything specified in the comments (Enabling Unity Audio, Audio Source in this script GameObject, AudioListener in a separate GameObject)

The Log information:

  • Call to FMOD API with result OK
  • [FMOD] OutputWASAPI::mixerThread : Starvation detected in WASAPI output buffer!

I’ve checked the sample rates (Unity & FMOD) and they match 48000.
I’ve also tried increasing the DSP buffer size with

        FMODUnity.RuntimeManager.CoreSystem.setDSPBufferSize(1024, 4);

but still nothing seems to work

Can’t seem to figure out the problem, could you provide some guidance?

8

It’s likely that there’s a mismatch between the number of channels being used by FMOD at the point the capture DSP is inserted into the DSP graph compared to the Audio Filter being used in Unity’s audio system. There’s a number of ways to go about solving this depending on whether the speaker modes are conforming to their respective system speaker modes (i.e. Unity’s speaker mode in Audio preferences, FMOD’s speaker mode for the target platform), or are instead set to specific speaker modes irrespective of this, but fundamentally they need to match or you’ll end up with artifacting like the high pitched clicking you’re experiencing.

Since the empty AudioSource will match Unity’s default speaker mode at Edit → Project Settings → Audio → Default Speaker Mode, you can do one of the following:

  • Change the Unity setting to match the number of channels you’re expecting from FMOD
  • Have the FMOD DSP downmix to match the Unity setting

For the latter, you should be able to grab Unity’s speaker mode using AudioSettings.GetConfiguration().speakerMode, and then change the FMOD DSP’s speaker mode by using DSP::setChannelFormat and setting source_speakermode to match it.

9

Thanks for sharing, with a few modifications to your code i finally got the result. Put this script somewhere in a root or a start scene, fmod audio will be captured into audio source, but only when Unity Recorder is on (Time.captureFramerate != 0).
But you need to toggle “Allow unsafe” in the Player settings for this to be compiled, or use jeff_fmod’s version of fmod read dsp callback, those changes are just for some performance to avoid array allocations.

using System;
using System.Collections.Generic;
using System.Runtime.CompilerServices;
using FMOD;
using FMODUnity;
using UnityEngine;
using Debug = UnityEngine.Debug;

namespace Recorder
{
    
    [RequireComponent(typeof(AudioSource))]
    public sealed class FmodRecorderHelper : MonoBehaviour
    {
        private readonly List<float> _buffer = new();
        private DSP_DESCRIPTION _dspDesc;
        private DSP _dsp;
        private bool _isRunning = false;
        private bool _isCreatedDsp = false;

        private static bool _isInitialized = false;
        
        private void Awake()
        {
            if (_isInitialized)
            {
                Destroy(this);
                return;
            }
            
            _isInitialized = true;
            DontDestroyOnLoad(this);
        }

        private void Reset()
        {
            var audioSource = GetComponent<AudioSource>();

            audioSource.spatialBlend = 0f;
            audioSource.reverbZoneMix = 0f;
        }

        private void OnDestroy()
        {
            RemoveDsp();
        }

        private void OnAudioFilterRead(float[] data, int channels)
        {
            if (!_isRunning) return;

            if (_buffer.Count >= data.Length)
            {
                // Copy from intermediate buffer into Unity's audio buffer
                for (int i = 0; i < data.Length; i++)
                {
                    data[i] = _buffer[i];
                }
                
                _buffer.RemoveRange(0, data.Length);
            }
        }

        private void Update()
        {
            if (Time.captureFramerate == 0)
            {
                if (_isRunning) RemoveDsp();
                return;
            }
            
            if (_isRunning) return;
            
            CreateDsp();
            
            RuntimeManager.CoreSystem.getMasterChannelGroup(out var group);
            if (group.hasHandle())
            {
                _isRunning = true;
                CHECK_RESULT(group.addDSP(0, _dsp));
            } 
        }

        private void CreateDsp()
        {
            if (_isCreatedDsp) return;
            _isCreatedDsp = true;
            
            _dspDesc = new DSP_DESCRIPTION
            {
                numinputbuffers = 1,
                numoutputbuffers = 1,
                read = (ref DSP_STATE dsp_state, IntPtr inbuffer, IntPtr outbuffer, uint length, int inchannels, ref int outchannels) =>
                {
                    if (length > 0)
                    {
                        unsafe
                        {
                            // Copy to buffer
                            var tmp = new Span<float>(inbuffer.ToPointer(), (int) length * inchannels);
                            for (int i = 0; i < tmp.Length; i++)
                            {
                                _buffer.Add(tmp[i]);
                            }
                            
                            // Silence FMOD output
                            tmp = new Span<float>(outbuffer.ToPointer(), (int) length * inchannels);
                            for (int i = 0; i < tmp.Length; i++)
                            {
                                tmp[i] = 0f;
                            }
                        }
                    }

                    return RESULT.OK;
                },
            };

            var result = RuntimeManager.CoreSystem.createDSP(ref _dspDesc, out _dsp);

            var unitySpeakerMode = AudioSettings.GetConfiguration().speakerMode; 
            var speakerMode = unitySpeakerMode switch
            {
                AudioSpeakerMode.Mono => SPEAKERMODE.MONO,
                AudioSpeakerMode.Stereo => SPEAKERMODE.STEREO,
                AudioSpeakerMode.Quad => SPEAKERMODE.QUAD,
                AudioSpeakerMode.Surround => SPEAKERMODE.SURROUND,
                AudioSpeakerMode.Mode5point1 => SPEAKERMODE._5POINT1,
                AudioSpeakerMode.Mode7point1 => SPEAKERMODE._7POINT1,
                AudioSpeakerMode.Prologic => SPEAKERMODE.DEFAULT,
                _ => SPEAKERMODE.DEFAULT,
            };

            int numChannels = unitySpeakerMode switch
            {
                AudioSpeakerMode.Mono => 1,
                AudioSpeakerMode.Stereo => 2,
                AudioSpeakerMode.Quad => 4,
                AudioSpeakerMode.Surround => 5,
                AudioSpeakerMode.Mode5point1 => 6,
                AudioSpeakerMode.Mode7point1 => 7,
                AudioSpeakerMode.Prologic => 2,
                _ => 2,
            }; 
            
            _dsp.setChannelFormat(CHANNELMASK.STEREO, numChannels, speakerMode);
            
            CHECK_RESULT(result);
        }

        private void RemoveDsp()
        {
            _isRunning = false;
            
            if (!_isCreatedDsp) return;
            _isCreatedDsp = false;
            
            CHECK_RESULT(RuntimeManager.CoreSystem.getMasterChannelGroup(out var group));
            CHECK_RESULT(group.removeDSP(_dsp));
            CHECK_RESULT(_dsp.release());
        }
        
        private static void CHECK_RESULT(RESULT result, [CallerLineNumber] int sourceLineNumber = 0)
        {
            if (result == RESULT.OK) return;
            Debug.LogError($"Call to FMOD API failed with result: \"{Error.String(result)}\" line: {sourceLineNumber}");
        }
    }
    
}
10

Hey there, thanks for figuring this one out :slight_smile: But could you further specify what exactly needs to be done to get this to work? I’m still struggling.

I added your provided Script to my project, added a new GameObject to my scene, attached the FmodRecorderHelper to that object and then simply started my game and hit Record in the Unity Recorder - still the resulting video doesn’t have any audio.

Is there anything else I have to do? Do I need to do something with the AudioSource that gets added to the Object? Some specific Audio Settings in the project? Some other scripts or a specific Recorder config? I tried a few things but couldn’t get it to record

11

I’m in the same boat as IrregularHuman, I would love to get this script working if you could give us some tips for how you setup the scene please.

12

Hi,

Assuming that Unity Audio is enabled, and an AudioListener is present in the scene, from what I can see it’s likely that the recorder’s framerate is set to variable and not constant. In variable mode, Time.captureFramerate and Time.captureDeltaTime appear to always be 0, so the rest of the code in MonoBehavior.Update() never occurs. I’d recommend removing the if (Time.captureFramerate == 0) check from Update() and seeing whether that resolves the issue for you, or swapping the recorder to a constant framerate.

13

Hi, sorry for the late reply. My previous posted version worked somehow, but sound was off or late very often anyway.

Here is an updated version that syncs with unity update and works well with variable/const framerate of Unity Recorder. One thing: you need to set Project Settings/Audio/Disable Unity Audio = false before entering playmode. Create GameObject with this component in runtime. Another catch is that migh be crashes, relaunch Unity will help. This is the most stable what i managed to get.

Here is the script:

using System;
using System.Collections.Generic;
using System.Runtime.InteropServices;
using System.Threading;
using Sirenix.OdinInspector;
using UnityEngine;

namespace _Project.Recorder
{
    
    internal sealed class FmodAudioRecorderDSP : MonoBehaviour
    {
        [Title("Diagnostics")]
        [Tooltip("Print capture stats and mixer mode changes to the console. Errors are reported regardless")]
        [SerializeField] private bool showLogs;

        [Tooltip("Log a stats line every N seconds while capturing. 0 - only the final summary")]
        [SerializeField] [MinValue(0f)] [ShowIf(nameof(showLogs))] private float statsLogInterval = 5f;

        [Title("Non-realtime capture")]
        [Tooltip("Switch FMOD to NOSOUND_NRT and advance its mixer by hand, one video frame worth of audio per rendered frame. " +
                 "FMOD goes silent while this is on - that is the point of offline rendering")]
        [SerializeField] private bool driveMixerManually = true;

        [Tooltip("Stay this many audio frames ahead of the video timeline so Unity never pulls an empty queue")]
        [SerializeField] [MinValue(0)] private int aheadFrames = 1024;

        [Tooltip("Safety cap on CoreSystem.update() calls per rendered frame")]
        [SerializeField] [MinValue(1)] private int maxPumpsPerFrame = 64;

        private FMOD.DSP_READ_CALLBACK mReadCallback;
        private FMOD.DSP mCaptureDSP;
        private GCHandle mObjHandle;

        private int mFrontBufferPosition = 0;
        private readonly Queue<float[]> mFullBufferQueue = new();
        private readonly Queue<float[]> mEmptyBufferQueue = new();
        private readonly object lockOb = new();

        private static int nbChannelDetected = 0;

        // Stats counters. Written from the FMOD mixer thread and from Unity's audio thread,
        // read on the main thread - every access goes through Interlocked, and nothing here logs.
        private long mProducedFrames;
        private long mConsumedFrames;
        private long mDeliveredFrames;
        private long mFmodCallbacks;
        private long mFilterReads;

        // Main thread only.
        private int mSampleRate;
        private int mRenderedFrames;
        private int mCaptureFrames;
        private double mTimelineFrames;
        private float mRealStartTime;
        private float mNextStatsLogTime;

        // Manual mixer pumping, main thread only.
        private FMOD.OUTPUTTYPE mPreviousOutput;
        private bool mOutputSwitched;
        private long mPumpCalls;
        private int mPumpCapHits;

        // Start, not OnEnable: to buffer between object creation and dsp start
        private void Start()
        {
            StartDSP();
        }

        private void OnDisable()
        {
            StopDSP();
        }

        private void StartDSP()
        {
            ResetStats();

            var config = AudioSettings.GetConfiguration();
            int unitySampleRate = config.sampleRate;
            int unityChannels = config.speakerMode == AudioSpeakerMode.Stereo ? 2 : (int) config.speakerMode;
            int fmodSampleRate;

            FMOD.SPEAKERMODE fmodSpeakerMode;
            FMODUnity.RuntimeManager.CoreSystem.getSoftwareFormat(out fmodSampleRate, out fmodSpeakerMode, out _);

            int fmodChannels = fmodSpeakerMode == FMOD.SPEAKERMODE.STEREO ? 2 :
                fmodSpeakerMode == FMOD.SPEAKERMODE.MONO ? 1 :
                0; // Default to 0 for unsupported speaker modes(e.g. Surround)

            string unityFormat = unityChannels == 1 ? "Mono" :
                unityChannels == 2 ? "Stereo" : "Unsupported";

            string fmodFormat = fmodChannels == 1 ? "Mono" :
                fmodChannels == 2 ? "Stereo" : "Unsupported";

            if (fmodSampleRate != unitySampleRate || fmodChannels != unityChannels)
            {
                Debug.LogError($"FMOD/Unity audio mismatch or unsupported channel format. Unity: {unitySampleRate}Hz/{unityFormat}, FMOD: {fmodSampleRate}Hz/{fmodFormat}\n" +
                               $"Please ensure FMOD and Unity use the same sample rate and channel layout (Mono or Stereo only).");
                enabled = false;
                return;
            }

            mSampleRate = unitySampleRate;

            // Switch the output before the tap is attached: setOutput tears the output down and brings it back up.
            EnterNonRealtimeMode();

            mReadCallback = CaptureDSPReadCallback;
            mObjHandle = GCHandle.Alloc(this);
            var desc = new FMOD.DSP_DESCRIPTION
            {
                numinputbuffers = 1,
                numoutputbuffers = 1,
                read = mReadCallback,
                userdata = GCHandle.ToIntPtr(mObjHandle),
            };

            // Attach custom DSP to master channel group
            if (FMODUnity.RuntimeManager.CoreSystem.getMasterChannelGroup(out var masterCG) == FMOD.RESULT.OK)
            {
                if (FMODUnity.RuntimeManager.CoreSystem.createDSP(ref desc, out mCaptureDSP) == FMOD.RESULT.OK)
                {
                    if (masterCG.addDSP(FMOD.CHANNELCONTROL_DSP_INDEX.TAIL, mCaptureDSP) == FMOD.RESULT.OK)
                        // channelmask устарел, FMOD его игнорирует и пишет об этом в консоль
                        mCaptureDSP.setChannelFormat(0, 2, FMOD.SPEAKERMODE.STEREO);
                    else
                        Debug.LogWarning("FMOD: Failed to add DSP to master channel group.");
                }
                else
                {
                    Debug.LogWarning("FMOD: Failed to create DSP.");
                }
            }
            else
            {
                Debug.LogWarning("FMOD: Failed to retrieve master channel group.");
            }
        }
        
        private void StopDSP()
        {
            LogStats("final");

            // Detach first, free the handle second. A live DSP whose userdata points at a freed
            // GCHandle is a hard crash on the next mixer callback, so the tap must go down
            // even when the master channel group cannot be reached.
            if (mCaptureDSP.hasHandle())
            {
                if (FMODUnity.RuntimeManager.CoreSystem.getMasterChannelGroup(out var masterCG) == FMOD.RESULT.OK)
                    masterCG.removeDSP(mCaptureDSP);
                else
                    Debug.LogError("FMOD: master channel group is unreachable, releasing the capture DSP anyway.");

                mCaptureDSP.release();
                mCaptureDSP = default;
            }

            if (mObjHandle.IsAllocated) mObjHandle.Free();

            ExitNonRealtimeMode();

            lock (lockOb)
            {
                mFullBufferQueue.Clear();
                mEmptyBufferQueue.Clear();
            }
        }

        #region Manual mixer pumping

        private void EnterNonRealtimeMode()
        {
            if (!driveMixerManually) return;

            var core = FMODUnity.RuntimeManager.CoreSystem;

            var result = core.getOutput(out mPreviousOutput);
            if (result != FMOD.RESULT.OK)
            {
                Debug.LogError($"FMOD: getOutput failed ({result}). Staying realtime - capture will drift on constant frame rate.");
                return;
            }

            result = core.setOutput(FMOD.OUTPUTTYPE.NOSOUND_NRT);
            if (result != FMOD.RESULT.OK)
            {
                Debug.LogError($"FMOD: setOutput(NOSOUND_NRT) failed ({result}). Staying realtime - capture will drift on constant frame rate.");
                return;
            }

            mOutputSwitched = true;

            if (showLogs) Debug.Log($"FMOD: mixer is driven manually, output {mPreviousOutput} -> NOSOUND_NRT.");
        }

        private void ExitNonRealtimeMode()
        {
            if (!mOutputSwitched) return;

            mOutputSwitched = false;

            var result = FMODUnity.RuntimeManager.CoreSystem.setOutput(mPreviousOutput);
            if (result != FMOD.RESULT.OK)
            {
                Debug.LogError($"FMOD: failed to restore output {mPreviousOutput} ({result}). " +
                               "Sound will stay silent until Play Mode is restarted.");
                return;
            }

            if (showLogs) Debug.Log($"FMOD: mixer is back on its own clock, output NOSOUND_NRT -> {mPreviousOutput}.");
        }

        // Runs after game logic has posted this frame's events, so the mixer renders what the frame actually asked for.
        private void LateUpdate()
        {
            if (!mOutputSwitched) return;
            if (mSampleRate <= 0) return;

            long target = (long) mTimelineFrames + aheadFrames;
            var core = FMODUnity.RuntimeManager.CoreSystem;

            int pumps = 0;
            while (Interlocked.Read(ref mProducedFrames) < target && pumps < maxPumpsPerFrame)
            {
                if (core.update() != FMOD.RESULT.OK) break;
                pumps++;
            }

            mPumpCalls += pumps;

            if (pumps >= maxPumpsPerFrame) mPumpCapHits++;
        }

        #endregion

        private void OnAudioFilterRead(float[] data, int channels)
        {
            // Audio thread: count only, never log.
            Interlocked.Increment(ref mFilterReads);
            Interlocked.Add(ref mConsumedFrames, channels > 0 ? data.Length / channels : 0);

            // Avoid leftover noise
            Array.Clear(data, 0, data.Length);

            lock (lockOb)
            {
                int offset = 0;
                while (mFullBufferQueue.Count > 0 && offset < data.Length)
                {
                    float[] front = mFullBufferQueue.Peek();

                    int remainingInFront = front.Length - mFrontBufferPosition;

                    if (remainingInFront <= 0)
                    {
                        mFullBufferQueue.Dequeue();
                        mFrontBufferPosition = 0;
                        continue;
                    }

                    int remainingInData = data.Length - offset;

                    int copyLength = Math.Min(remainingInFront, remainingInData);
                    Array.Copy(front, mFrontBufferPosition, data, offset, copyLength);

                    mFrontBufferPosition += copyLength;
                    offset += copyLength;

                    // If buffer fully consumed, recycle it
                    if (mFrontBufferPosition < front.Length) continue;

                    mFullBufferQueue.Dequeue();
                    mFrontBufferPosition = 0;

                    // Recycle consumed buffers, limit to 32 stored
                    if (mEmptyBufferQueue.Count < 32) mEmptyBufferQueue.Enqueue(front);
                }

                Interlocked.Add(ref mDeliveredFrames, channels > 0 ? offset / channels : 0);
            }
        }

        [AOT.MonoPInvokeCallback(typeof(FMOD.DSP_READ_CALLBACK))]
        private static FMOD.RESULT CaptureDSPReadCallback(ref FMOD.DSP_STATE dsp_state, IntPtr inbuffer, IntPtr outbuffer, uint length, int inchannels, ref int outchannels)
        {
            IntPtr userData;

            dsp_state.functions.getuserdata(ref dsp_state, out userData);
            var objHandle = GCHandle.FromIntPtr(userData);
            var obj = objHandle.Target as FmodAudioRecorderDSP;
            if (inchannels > nbChannelDetected)
                nbChannelDetected = inchannels;
            if (inchannels > 2)
            {
                Debug.LogError("Channels FMOD > 2!");
                inchannels = 2;
            }

            // FMOD mixer thread: count only, never log.
            Interlocked.Increment(ref obj.mFmodCallbacks);
            Interlocked.Add(ref obj.mProducedFrames, length);

            int lengthElements = (int) length * inchannels;
            float[] buffer;

            // Try to reuse a managed buffer of the exact size to reduce GC pressure.
            lock (obj.lockOb)
            {
                if (obj.mEmptyBufferQueue.Count > 0)
                {
                    float[] tmp = obj.mEmptyBufferQueue.Dequeue();
                    buffer = tmp.Length == lengthElements ? tmp : new float[lengthElements];
                }
                else
                {
                    buffer = new float[lengthElements];
                }
            }

            Marshal.Copy(inbuffer, buffer, 0, lengthElements);

            lock (obj.lockOb)
            {
                obj.mFullBufferQueue.Enqueue(buffer);
            }

            // Pass through to FMOD downstream (so monitoring still works)
            Marshal.Copy(buffer, 0, outbuffer, lengthElements);
            outchannels = inchannels;
            return FMOD.RESULT.OK;
        }

        #region Stats

        private void Update()
        {
            mRenderedFrames++;

            // The video timeline: a rendered frame is worth exactly one frame delta of audio.
            float delta = Time.captureDeltaTime;
            if (delta > 0f) mCaptureFrames++;
            else delta = Time.unscaledDeltaTime;

            mTimelineFrames += (double) delta * mSampleRate;

            if (statsLogInterval <= 0f) return;
            if (Time.realtimeSinceStartup < mNextStatsLogTime) return;

            mNextStatsLogTime = Time.realtimeSinceStartup + statsLogInterval;
            LogStats("progress");
        }

        // Explicit request, so it prints even with showLogs off.
        [Button("Log stats now")]
        private void LogStatsNow()
        {
            LogStats("manual", force: true);
        }

        private void ResetStats()
        {
            Interlocked.Exchange(ref mProducedFrames, 0L);
            Interlocked.Exchange(ref mConsumedFrames, 0L);
            Interlocked.Exchange(ref mDeliveredFrames, 0L);
            Interlocked.Exchange(ref mFmodCallbacks, 0L);
            Interlocked.Exchange(ref mFilterReads, 0L);

            mRenderedFrames = 0;
            mCaptureFrames = 0;
            mTimelineFrames = 0d;
            mPumpCalls = 0L;
            mPumpCapHits = 0;
            mRealStartTime = Time.realtimeSinceStartup;
            mNextStatsLogTime = Time.realtimeSinceStartup + statsLogInterval;
        }

        // Main thread only - the mixer and audio threads never log, they only bump counters.
        private void LogStats(string title, bool force = false)
        {
            if (!showLogs && !force) return;
            if (Interlocked.Read(ref mFmodCallbacks) == 0L && Interlocked.Read(ref mFilterReads) == 0L) return;

            Debug.Log(BuildStatsReport(title));
        }

        private string BuildStatsReport(string title)
        {
            long produced = Interlocked.Read(ref mProducedFrames);
            long consumed = Interlocked.Read(ref mConsumedFrames);
            long delivered = Interlocked.Read(ref mDeliveredFrames);
            long fmodCallbacks = Interlocked.Read(ref mFmodCallbacks);
            long filterReads = Interlocked.Read(ref mFilterReads);

            long expected = (long) mTimelineFrames;
            float timelineElapsed = FramesToSeconds(expected);
            float realElapsed = Time.realtimeSinceStartup - mRealStartTime;
            float clockRatio = timelineElapsed > 0f ? realElapsed / timelineElapsed : 0f;

            float captureDelta = Time.captureDeltaTime;
            string frameRateMode = captureDelta > 0f ? $"{1f / captureDelta:F1} fps, constant" : "variable frame rate";
            string mixerMode = mOutputSwitched ? "NOSOUND_NRT, driven by hand" : "realtime, own clock";

            int queuedBuffers;
            lock (lockOb) queuedBuffers = mFullBufferQueue.Count;

            return $"[FmodAudioRecorderDSP] stats ({title})\n" +
                   $"  render    : {mRenderedFrames} frames ({mCaptureFrames} captured), captureDeltaTime {captureDelta:F5}s ({frameRateMode})\n" +
                   $"  clock     : timeline {timelineElapsed:F2}s, real {realElapsed:F2}s, real/timeline {clockRatio:F2}\n" +
                   $"  mixer     : {mixerMode}, {mPumpCalls} pumps, {PumpsPerFrame():F2} per rendered frame, cap hits {mPumpCapHits}\n" +
                   $"  produced  : {produced} frames / {FramesToSeconds(produced):F2}s (FMOD mixer, {fmodCallbacks} callbacks)\n" +
                   $"  consumed  : {consumed} frames / {FramesToSeconds(consumed):F2}s (Unity pulled, {filterReads} reads)\n" +
                   $"  delivered : {delivered} frames / {FramesToSeconds(delivered):F2}s, underrun {consumed - delivered} frames\n" +
                   $"  expected  : {expected} frames / {timelineElapsed:F2}s (video timeline x {mSampleRate}Hz)\n" +
                   $"  drift     : produced-expected {produced - expected} frames / {FramesToSeconds(produced - expected):F2}s\n" +
                   $"  backlog   : {produced - delivered} frames / {FramesToSeconds(produced - delivered):F2}s, queue {queuedBuffers} buffers";
        }

        private float FramesToSeconds(long frames)
        {
            return mSampleRate > 0 ? frames / (float) mSampleRate : 0f;
        }

        private float PumpsPerFrame()
        {
            return mRenderedFrames > 0 ? mPumpCalls / (float) mRenderedFrames : 0f;
        }

        #endregion
    }
    
}
14

Hi, checkout the last post, i got some better results with the new script

15

Hi, here is the last post with the new script, hope it will help