Trying to get the waveform/envelope of an EventInstance but Sound.ReadData failing

Unity
,
1

Hi,

I would like to display the waveform (or envelope) of an fmod event before playing it, so that the user can see what sound it is beforehand.

Thanks to this Topic, I think I’m getting close: Trouble visualizing waveform from event without separate Sound object - Unity - FMOD Forums

Everything works fine except that Sound.ReadData() returns an ERR_UNSUPPORTED error.

Here is my code (most is copy pasted from the given topic, it’s just here for context):

public EventReference eventRef;
public void GetSound()
{
    eventInstance = RuntimeManager.CreateInstance(eventRef);
    //For now I'm starting it so that it loads, but I intend to either not start it or stop it right after as I don't want to play it now (I just need to readData)
    eventInstance.start();
    StartCoroutine(GetSoundFromInstance(eventInstance));
}

IEnumerator GetSoundFromInstance(EventInstance eventInstance)
{
    if (eventInstance.hasHandle() && !sound.hasHandle())
    {
        //Wait for the sound to load
        FMOD.Studio.PLAYBACK_STATE state;
        FMOD.RESULT result;
        do
        {
            result = eventInstance.getPlaybackState(out state);
            if (result == FMOD.RESULT.OK)
            {
                if (state == FMOD.Studio.PLAYBACK_STATE.PLAYING)
                {
                    if (eventInstance.getChannelGroup(out FMOD.ChannelGroup group) == FMOD.RESULT.OK)
                    {
                        sound = FindCurrentSound(group);
                        sound.getName(out string name, 256);
                        //This works fine for eg, as all others getSomething()
                        UnityEngine.Debug.Log($"Audio Name: {name}");

                        sound.getLength(out uint length, TIMEUNIT.RAWBYTES);
                        byte[] buffer = new byte[length];
                        //This seems correct as well
                        UnityEngine.Debug.Log($"Sound Length: {length}");

                        //================ISSUE IS HERE========================
                        FMOD.RESULT resultRead = sound.readData(buffer, out uint read);

                        if (resultRead != FMOD.RESULT.OK)
                        {
                            //This logs ERR_UNSUPPORTED
                            UnityEngine.Debug.Log($"Failed to retrieve data from sound: {resultRead }");
                        }

                        //Do things with buffer...
                    }
                }
            }
            //This is always waited once, as the sound seem to take a frame to start playing
            yield return null;
        } while (result != RESULT.OK || state != FMOD.Studio.PLAYBACK_STATE.PLAYING);
    }
}

//This is pasted from the above topic, seems to work fine
FMOD.Sound FindCurrentSound(FMOD.ChannelGroup channelGroup)
{
    FMOD.Sound sound = new FMOD.Sound();

    // Check if we are currently in the right channel group
    if (channelGroup.getNumChannels(out int numChannels) == FMOD.RESULT.OK && numChannels > 0)
    {
        if (numChannels > 1)
        {
            UnityEngine.Debug.Log("More than 1 channel");
        }
        else
        {
            if (channelGroup.getChannel(0, out FMOD.Channel channel) == FMOD.RESULT.OK)
            {
                if (channel.getCurrentSound(out sound) == FMOD.RESULT.OK)
                {
                    UnityEngine.Debug.Log("Found the correct Sound");
                }
                else
                {
                    UnityEngine.Debug.Log("Could not retrieve sound");
                    sound = new FMOD.Sound();
                }
            }
        }
    }

    // Check if the sound has been found
    if (!sound.hasHandle())
    {
        // If not, continue the recusion
        channelGroup.getName(out string name, 256);
        UnityEngine.Debug.Log(name);

        if (channelGroup.getNumGroups(out int numGroups) == FMOD.RESULT.OK && numGroups > 0)
        {
            if (numGroups > 1)
            {
                UnityEngine.Debug.Log("More than one Group");
            }
            else
            {
                if (channelGroup.getGroup(0, out FMOD.ChannelGroup child) == FMOD.RESULT.OK)
                {
                    sound = FindCurrentSound(child);
                }
            }
        }
    }
    // Else the sound has been found so return the sound
    // This will either be a real sound or not depending if it was found in the recussion.
    return sound;
}

I guess I’m missing a call to something but can’t figure what. Any idea?

2

Unfortunately, while it’s fairly easy to grab sample data and visualize a waveform on the fly, the Studio API is not really built to be able to grab a sound’s entire waveform in a convenient way.

There’s three primary ways to load a sound:

  1. As a stream
  2. As compressed sample data
  3. As decompressed sample data

In the case of the latter two, the file is opened, the sample data copied into memory, and the file closed. Sound.readData() requires the file to still be opened, which for a stream it is, but for the other modes it isn’t.

When Sound.readData() is called on a sound being played by an event like you’re doing, if the sound is a stream, the sample data isn’t easily available in memory since it’s being streamed from the file, meaning that Sound.readData will compete with FMOD’s streaming process. If the sound is compressed or decompressed samples, the sample data will have been copied to memory and the file itself closed already, causing readData to return ERR_UNSUPPORTED like you’re seeing.

It should be possible to use Sound.lock() and Sound.unlock() to grab the entirety of a non-streaming sound’s sample data, but this comes with caveats - if the sound’s sample data has been loaded in the compressed mode, Studio’s default, Sound.lock() will provide you with the sample data still compressed, which you will not be able to generate a waveform from without decompressing.

As a result, you may find it easier to process the event ahead of time to generate a waveform, either at runtime but before playing any of the events in question, or during the development process as image assets in your Unity project.

3

Thanks for your answer.

I did exactly what you suggest in the meantime and it works for us.

Here is my code if anyone looking to do the same thing in the future. Usage: Put it on any GameObject in an empty scene with an FmodListener, add whatever events you want to the list, then hit Play. It will play all sounds one after the other and save their waveforms as scriptable objects (you gotta create your scriptableObject with a float array)

using UnityEngine;
using System.Collections.Generic;
using UnityEditor;
using System.Collections;
using FMODUnity;

namespace EDream.Waveform
{
    public class WaveformSignalBaking : MonoBehaviour
    {
        [Header("Data")]
        public List<EventReference> eventToBake;

        [Header("Baking Process")]
        FMODCaptureDSP capture;
        public List<float> audioData;

        void Start()
        {
            capture = GetComponent<FMODCaptureDSP>();
            audioData = new List<float>();

            StartCoroutine(BakeAllSounds());
        }

        IEnumerator BakeAllSounds()
        {
            WaveformData[] data = new WaveformData[eventToBake.Count];
            for (int i = 0; i < eventToBake.Count; i++)
            {
                audioData.Clear();
                //Process the sound of this id
                yield return StartCoroutine(PlayEventAndReadData(eventToBake[i]));
            }
        }

        IEnumerator PlayEventAndReadData(EventReference eventReference)
        {
            //Empty wait to let space in between sound recordings so Unity can clear things up from previous sound
            yield return null;

            RuntimeManager.StudioSystem.getEvent(eventReference.Path, out FMOD.Studio.EventDescription eventDescription);
            eventDescription.createInstance(out FMOD.Studio.EventInstance eventInstance);
            eventInstance.set3DAttributes(RuntimeUtils.To3DAttributes(gameObject));
            eventInstance.setParameterByName("PersoGain", 1);
            eventInstance.start();

            float startTime = Time.realtimeSinceStartup;
            capture.OnNewAudioData += OnNewAudioData;

            //Wait for the sound to start
            FMOD.Studio.PLAYBACK_STATE playbackState;
            do
            {
                eventInstance.getPlaybackState(out playbackState);
                yield return null;
            }
            while (playbackState != FMOD.Studio.PLAYBACK_STATE.PLAYING);

            //While sound is playing
            while (playbackState == FMOD.Studio.PLAYBACK_STATE.PLAYING)
            {
                eventInstance.getPlaybackState(out playbackState);
                yield return null;
            }

            float timeLength = Time.realtimeSinceStartup - startTime;
            capture.OnNewAudioData -= OnNewAudioData;

            CreateScriptableDataObj(WaveformDatabase.GetNameFromEventRef(in eventReference), audioData.ToArray(), timeLength);

            yield return null;
        }

        private void OnNewAudioData(float[] data)
        {
            audioData.AddRange(data);
        }

        void CreateScriptableDataObj(string eventName, float[] audioData, float timeLength)
        {
            ScriptableWaveformData asset = ScriptableObject.CreateInstance<ScriptableWaveformData>();
            asset.waveData = audioData;

            string filePath = "Assets/Resources/"+eventName+".asset";
            AssetDatabase.CreateAsset(asset, filePath);
            AssetDatabase.SaveAssets();
            Debug.Log($"<color=#0DD3B6>Baked {eventName} signal at {filePath} sucessfully!</color>");
        }
    }
}
4

Happy to hear!