Unity Recorder script (from Scripting Examples) has few seconds of delay

Hi,

I’m trying to record gameplay with the Unity Recorder. I was pleased to see that there is now a scripting example that should allow to record the sound from FMOD. It does record sound, but it’s very much delayed (Example - the light dots should match the sound).

Here is what I did:

  1. Copy-pasted the code from here in a script in my project (Simply renamed the class)
  2. Added it to an empty GameObject in my scene
  3. Added an AudioListener to the same gameObject
  4. Enabled Unity Audio
  5. Set the Unity Audio Sample Rate to 48.000 (same as FMOD audio)
  6. Hit play
  7. I can hear the sound in double as planned, tho one of them is delayed by ~1 sec
  8. Started recording with Unity Recorder. In the video output, sound is delayed by a few seconds.

Settings:

I also tried the script on a different project on a different computer, same result.

Am I missing something, or is the script broken?

As a side note, the fact that FMOD struggles to work with Unity’s Recorder is a very big deal for us. I know that it’s 50% from your side and 50% from Unity’s side, but I believe that two tools that are both used hand in hand for a large portion of their users could figure something out together…

We figured out the problem!
The issue was that our FMOD output was using 6 channels, and Unity Audio was expecting 2.
As stated in the script, FMOD must be in Stereo (2 channels) and not Surround something. BUT it appears this must be done in FMOD Studio→Preferences→Build→Stereo. Note that in Unity→FMODSettings there is a similar setting but it doesn’t appear to work for this issue.

Happy to hear that you managed to resolve the issue!

As far as I’m aware, the internal speakermode of the FMOD mixer shouldn’t make any impact on the recording latency, since the capture DSP that’s inserted into the mixer is set up so that incoming audio is downmixed to stereo. It more likely that there’s some interaction between Unity/FMOD’s audio buffer sizes, and the Unity recorder.

If you’re able to reproduce the issue again, could I get your exact Unity and FMOD for Unity versions, as well as screenshots of your settings for the Unity Recorder and ScriptUsageUnityRecorder component?

Here are our current settings:

I tried reverting back to surround 5.1 in the FMOD studio settings (then some other FMOD settings in Unity) I couldn’t reproduce the issue we had before so idk. Tho we did try many times with different settings, including the ones we have now, before switching to stereo in FMOD studio which immediatly fixed the issue. Maybe rebuilding the banks simply triggered a refresh somewhere which was needed for some setting to take effect or something.

Either way, we tried recording FMOD audio output directly using the script given by stenjus in this other thread: Can't record with Unity Recorder - #6 by stenjus (we used it simply to get the .wav file and got rid of the video and ffmpeg stuff). We could confirm at this point that our audio was using 6 channels, while the ScriptUsageUnityRecorder is supposed to work for stereo only. When we went down to 2 channels (we assumed thanks to the change in fmod studio), the script worked correctly.

Here is our version of ScriptUsageUnityRecorder (copy pasted, simply renamed I think):

//--------------------------------------------------------------------
//
// This is a Unity behaviour script that demonstrates how to capture
// FMOD's mixed audio output by creating a custom DSP and attaching it to
// the master channel group.
//
// The captured audio is then routed to Unity's OnAudioFilterRead callback
// so that it can be recorded by Unity Recorder.
//
// Steps to use:
// 1. Ensure Unity audio is enabled.
// 2. Attach this script to a GameObject that has an active AudioListener.
// 3. Ensure FMOD and Unity use the same sample rate and channel format (Mono or Stereo only).
//    Unity Recorder does not support channel formats above stereo (e.g., 5.1, 7.1).
//
// NOTE: In Editor Play (not recording) you may hear double monitoring:
// FMOD pass-through + Unity OnAudioFilterRead copy. This is expected.
// To avoid: temporarily mute the FMOD Master or route this GameObject to a silent Mixer.
// With Unity Recorder, the Listener is usually muted so double monitoring won't occur.
//
// This document assumes familiarity with Unity scripting. See
// https://unity3d.com/learn/tutorials/topics/scripting for resources
// on learning Unity scripting.
//
//--------------------------------------------------------------------

using System;
using System.Collections.Generic;
using System.Runtime.InteropServices;
using UnityEngine;

#if UNITY_EDITOR
public class FMODRecorderHelper : MonoBehaviour
{
    private FMOD.DSP_READ_CALLBACK mReadCallback;
    private FMOD.DSP mCaptureDSP;
    private GCHandle mObjHandle;
    [Tooltip("Wait this many buffers before feeding Unity to avoid start noise")]
    [SerializeField, Min(0)]
    private int warmupBufferCount = 2;
    private int mFrontBufferPosition = 0;
    private Queue<float[]> mFullBufferQueue = new();
    private Queue<float[]> mEmptyBufferQueue = new();
    private readonly object lockOb = new();

    public static int nbChannelDetected = 0;

    void Start()
    {
        // Prevent FMOD DSP initialization when not in Play Mode.
        if (!Application.isPlaying) return;
        // Validate Unity and FMOD audio config match
        var config = AudioSettings.GetConfiguration();
        int unitySampleRate = config.sampleRate;
        int unityChannels = config.speakerMode == AudioSpeakerMode.Stereo ? 2 : (int)config.speakerMode;
        int fmodSampleRate;

        FMOD.SPEAKERMODE fmodSpeakerMode;
        FMODUnity.RuntimeManager.CoreSystem.getSoftwareFormat(out fmodSampleRate, out fmodSpeakerMode, out _);

        int fmodChannels = fmodSpeakerMode == FMOD.SPEAKERMODE.STEREO ? 2 :
                           fmodSpeakerMode == FMOD.SPEAKERMODE.MONO ? 1 :
                           0; // Default to 0 for unsupported speaker modes(e.g. Surround)

        string unityFormat = unityChannels == 1 ? "Mono" :
                             unityChannels == 2 ? "Stereo" : "Unsupported";

        string fmodFormat = fmodChannels == 1 ? "Mono" :
                            fmodChannels == 2 ? "Stereo" : "Unsupported";

        if (fmodSampleRate != unitySampleRate || fmodChannels != unityChannels)
        {
            Debug.LogError($"FMOD/Unity audio mismatch or unsupported channel format. Unity: {unitySampleRate}Hz/{unityFormat}, FMOD: {fmodSampleRate}Hz/{fmodFormat}\n" +
                           $"Please ensure FMOD and Unity use the same sample rate and channel layout (Mono or Stereo only).");
            enabled = false;
            return;
        }
        mReadCallback = CaptureDSPReadCallback;
        mObjHandle = GCHandle.Alloc(this);
        var desc = new FMOD.DSP_DESCRIPTION
        {
            numinputbuffers = 1,
            numoutputbuffers = 1,
            read = mReadCallback,
            userdata = GCHandle.ToIntPtr(mObjHandle)
        };

        // Attach custom DSP to master channel group
        if (FMODUnity.RuntimeManager.CoreSystem.getMasterChannelGroup(out var masterCG) == FMOD.RESULT.OK)
        {
            if (FMODUnity.RuntimeManager.CoreSystem.createDSP(ref desc, out mCaptureDSP) == FMOD.RESULT.OK)
            {
                if (masterCG.addDSP(FMOD.CHANNELCONTROL_DSP_INDEX.TAIL, mCaptureDSP) == FMOD.RESULT.OK)
                {
                    mCaptureDSP.setChannelFormat(FMOD.CHANNELMASK.STEREO, 2, FMOD.SPEAKERMODE.STEREO);
                }
                else
                {
                    Debug.LogWarning("FMOD: Failed to add DSP to master channel group.");
                }
            }
            else
            {
                Debug.LogWarning("FMOD: Failed to create DSP.");
            }
        }
        else
        {
            Debug.LogWarning("FMOD: Failed to retrieve master channel group.");
        }
    }

    [AOT.MonoPInvokeCallback(typeof(FMOD.DSP_READ_CALLBACK))]
    static FMOD.RESULT CaptureDSPReadCallback(ref FMOD.DSP_STATE dsp_state, IntPtr inbuffer, IntPtr outbuffer, uint length, int inchannels, ref int outchannels)
    {
        var functions = dsp_state.functions;
        functions.getuserdata(ref dsp_state, out var userData);
        var objHandle = GCHandle.FromIntPtr(userData);
        var obj = objHandle.Target as FMODRecorderHelper;
        if (inchannels > nbChannelDetected)
            nbChannelDetected = inchannels;
        if (inchannels > 2)
        {
            Debug.LogError("Channels FMOD > 2!");
            inchannels = 2;
        }
        int lengthElements = (int)length * inchannels;
        float[] buffer;

        // Try to reuse a managed buffer of the exact size to reduce GC pressure.
        lock (obj.lockOb)
        {
            if (obj.mEmptyBufferQueue.Count > 0)
            {
                var tmp = obj.mEmptyBufferQueue.Dequeue();
                buffer = (tmp.Length == lengthElements) ? tmp : new float[lengthElements];
            }
            else
            {
                buffer = new float[lengthElements];
            }
        }

        Marshal.Copy(inbuffer, buffer, 0, lengthElements);

        lock (obj.lockOb)
        {
            obj.mFullBufferQueue.Enqueue(buffer);
        }

        // Pass through to FMOD downstream (so monitoring still works)
        Marshal.Copy(buffer, 0, outbuffer, lengthElements);
        outchannels = inchannels;
        return FMOD.RESULT.OK;
    }

    void OnDestroy()
    {
        if (!Application.isPlaying) return;

        if (mObjHandle.IsAllocated)
        {
            if (FMODUnity.RuntimeManager.CoreSystem.getMasterChannelGroup(out var masterCG) == FMOD.RESULT.OK && mCaptureDSP.hasHandle())
            {
                masterCG.removeDSP(mCaptureDSP);
                mCaptureDSP.release();
            }
            mObjHandle.Free();
        }

        lock (lockOb)
        {
            mFullBufferQueue.Clear();
            mEmptyBufferQueue.Clear();
        }
    }

    void OnAudioFilterRead(float[] data, int channels)
    {
        // Avoid leftover noise
        Array.Clear(data, 0, data.Length);

        lock (lockOb)
        {
            // Wait for a few captured blocks to avoid initial glitches/pops.
            if (mFullBufferQueue.Count > warmupBufferCount)
            {
                int offset = 0;
                while (mFullBufferQueue.Count > 0 && offset < data.Length)
                {
                    float[] front = mFullBufferQueue.Peek();

                    int remainingInFront = front.Length - mFrontBufferPosition;

                    if (remainingInFront <= 0)
                    {
                        mFullBufferQueue.Dequeue();
                        mFrontBufferPosition = 0;
                        continue;
                    }

                    int remainingInData = data.Length - offset;

                    int copyLength = Math.Min(remainingInFront, remainingInData);
                    Array.Copy(front, mFrontBufferPosition, data, offset, copyLength);

                    mFrontBufferPosition += copyLength;
                    offset += copyLength;

                    // If buffer fully consumed, recycle it
                    if (mFrontBufferPosition >= front.Length)
                    {
                        mFullBufferQueue.Dequeue();
                        mFrontBufferPosition = 0;

                        // Recycle consumed buffers, limit to 32 stored
                        if (mEmptyBufferQueue.Count < 32)
                        {
                            mEmptyBufferQueue.Enqueue(front);
                        }
                    }

                }
            }
        }
    }
}
#endif

Unity recorder:

Also if I can help someone, we made this simple custom window (probably not that clean yet) to quickly add the ScriptUsageUnityRecorder & AudioListener in game when we want to record:

using System;
using System.IO;
using UnityEditor;
using UnityEditor.Presets;
using UnityEditor.Recorder;
using UnityEngine;

namespace Recording
{
    /// <summary>
    /// Usage: Create a preset from the Unity Recorder with the desired encoding settings (top left of Unity recorder's window).
    /// Name it MovieRecorderSettings.preset and save it anywhere.
    /// Open this window (Topbar->Tools) and start recording from there.
    /// The prepare button will add the AudioListener and the FMOD script needed to send the audio to the Unity Recorder.
    /// Starting and stopping the record are simply convenient ways to interact with the Unity Recorder from the same window.
    /// </summary>
    public class VideoRecorderWindow : EditorWindow
    {
        RecorderController recorderController;
        public Preset preset;
        bool isPrepared;
        bool isRecording;

        public int take;
        const string OUTPUT_FOLDER = "Recordings";

        void OnInspectorUpdate()
        {
            // Call Repaint on OnInspectorUpdate as it repaints the windows
            // less times as if it was OnGUI/Update
            Repaint();
        }

        public void PrepareFMODRecorder()
        {
            //Generate the FMOD Recorder helper obj
            GameObject helper = new GameObject("FMOD Recorder Helper", typeof(ScriptUsageUnityRecorder), typeof(AudioListener));

            // Find the preset asset named "MovieRecorderSettings" anywhere in the project and load it
            string[] guids = AssetDatabase.FindAssets("MovieRecorderSettings t:Preset");
            if (guids == null || guids.Length == 0)
            {
                Debug.LogError("MovieRecorderSettings.preset not found in project assets.");
                return;
            }
            string path = AssetDatabase.GUIDToAssetPath(guids[0]);
            preset = AssetDatabase.LoadAssetAtPath<Preset>(path);

            isPrepared = true;
            isRecording = false;
        }

        public void StartRecorder()
        {
            if (!isPrepared)
                return;

            // Create RecorderSettings from the provided Preset path.
            RecorderSettings recorderSettings = LoadRecorderSettingsFromPreset(preset);
            recorderSettings.FrameRate = 60;
            recorderSettings.Take = take;

            // Create a new RecorderControllerSettings to set the start and end frame for
            // the recording session and add the RecorderSettings to it.
            var controllerSettings = ScriptableObject.CreateInstance<RecorderControllerSettings>();
            controllerSettings.AddRecorderSettings(recorderSettings);
            controllerSettings.SetRecordModeToManual();

            // Create and setup a new RecorderController and start the recording.
            recorderController = new RecorderController(controllerSettings);
            recorderController.PrepareRecording();
            recorderController.StartRecording();

            isRecording = true;
        }

        public void StopRecorder()
        {
            if (!isRecording)
                return;

            recorderController.StopRecording();
            isRecording = false;
            take++;
        }

        static RecorderSettings LoadRecorderSettingsFromPreset(Preset _preset)
        {
            // Create a new RecorderSettings instance and apply the Preset to it.
            RecorderSettings outSettings = (RecorderSettings)ScriptableObject.CreateInstance(typeof(MovieRecorderSettings));

            _preset.ApplyTo(outSettings);
            outSettings.name = _preset.name;

            return outSettings;
        }

        void OpenRecordingsFolder(string folder)
        {
            //Check if folder exists, try creating it otherwise
            if (!Directory.Exists(folder))
            {
                try
                {
                    Directory.CreateDirectory(folder);
                }
                catch (Exception e)
                {
                    Debug.LogError($"Failed to create folder {folder}: {e.Message}");
                    return;
                }
            }

            // Use Editor utility to reveal in OS file explorer
            EditorUtility.RevealInFinder(folder + "/");
        }

        [MenuItem("Window/Custom/CustomVideoRecorder")]
        public static void ShowWindow()
        {
            VideoRecorderWindow wnd = GetWindow<VideoRecorderWindow>();
            wnd.titleContent = new GUIContent("Custom Video Recorder");
        }

        [Obsolete("Seems to crash sometimes/not work for recording when done in play mode")]
        private static void SetUnityAudio(bool value)
        {
            const string AudioSettingsAssetPath = "ProjectSettings/AudioManager.asset";
            SerializedObject audioManager = new SerializedObject(UnityEditor.AssetDatabase.LoadAllAssetsAtPath(AudioSettingsAssetPath)[0]);
            SerializedProperty property = audioManager.FindProperty("m_DisableAudio");

            property.boolValue = !value;

            audioManager.ApplyModifiedProperties();
        }

        private static bool GetUnityAudionEnabledState()
        {
            const string AudioSettingsAssetPath = "ProjectSettings/AudioManager.asset";
            SerializedObject audioManager = new SerializedObject(UnityEditor.AssetDatabase.LoadAllAssetsAtPath(AudioSettingsAssetPath)[0]);
            SerializedProperty property = audioManager.FindProperty("m_DisableAudio");

            return property.boolValue;
        }

        void OnGUI()
        {
            Color baseColor = GUI.color;
            var centered = new GUIStyle(EditorStyles.boldLabel) { alignment = TextAnchor.MiddleCenter };

            EditorGUILayout.Space(20);
            EditorGUILayout.LabelField("Video recorder", centered, GUILayout.ExpandWidth(true));
            EditorGUILayout.Space(20);

            if (GetUnityAudionEnabledState())
            {
                GUI.color = Color.yellow;
                EditorGUILayout.LabelField("You must remove the option", centered, GUILayout.ExpandWidth(true));
                EditorGUILayout.LabelField("Project Settings - Audio - Disable Unity Audio", centered, GUILayout.ExpandWidth(true));
                isPrepared = false;
                GUI.color = baseColor;
                return;
            }

            if (!Application.isPlaying)
            {
                GUI.color = Color.yellow;
                EditorGUILayout.LabelField("Game must be started", centered, GUILayout.ExpandWidth(true));
                isPrepared = false;
                GUI.color = baseColor;
            }
            else
            {
                if (!isPrepared)
                {
                    GUILayout.BeginHorizontal();
                    GUILayout.FlexibleSpace();
                    if (GUILayout.Button("Prepare Recording!", GUILayout.Width(180), GUILayout.Height(28)))
                        PrepareFMODRecorder();
                    GUILayout.FlexibleSpace();
                    GUILayout.EndHorizontal();
                }
                else
                {
                    EditorGUILayout.LabelField("All is well!", centered, GUILayout.ExpandWidth(true));
                }
            }

            EditorGUI.BeginDisabledGroup(!Application.isPlaying || !isPrepared || isRecording); // disable start when recording or not playing
            GUILayout.BeginHorizontal();
            GUILayout.FlexibleSpace();
            if (GUILayout.Button("Start Recording!", GUILayout.Width(180), GUILayout.Height(28)))
                StartRecorder();
            GUILayout.FlexibleSpace();
            GUILayout.EndHorizontal();
            EditorGUI.EndDisabledGroup();

            EditorGUI.BeginDisabledGroup(!Application.isPlaying || !isPrepared || !isRecording); // disable stop when not recording or not playing
            GUILayout.BeginHorizontal();
            GUILayout.FlexibleSpace();
            if (GUILayout.Button("Stop Recording!", GUILayout.Width(180), GUILayout.Height(28)))
                StopRecorder();
            GUILayout.FlexibleSpace();
            GUILayout.EndHorizontal();
            EditorGUI.EndDisabledGroup();

            EditorGUILayout.Space(20);
            EditorGUILayout.LabelField($"Folder: {Application.dataPath}/{OUTPUT_FOLDER}", centered, GUILayout.ExpandWidth(true));

            // Open folder button
            GUILayout.Space(10);
            GUILayout.BeginHorizontal();
            GUILayout.FlexibleSpace();
            if (GUILayout.Button("Open Recordings Folder", GUILayout.Width(220)))
            {
                OpenRecordingsFolder(Application.dataPath + "/../" + OUTPUT_FOLDER);
            }
            GUILayout.FlexibleSpace();
            GUILayout.EndHorizontal();

            GUI.color = baseColor;
        }
    }
}