Trouble visualizing waveform from event without separate Sound object

Hello, I am very new to FMOD Engine so apologies and thanks in advance.

I am trying to create a Unity game that is basically a mini DAW or a very simplified DAW.
Players should be able to visualize a track’s waveform as well as manipulate the audio in certain ways.

Everywhere online I see visualizations that update every frame, using a FFT DSP to display spectrum data as the audio plays.

What I need is to create a 2D waveform of the sound/track/event, and be able to use that event for further manipulations/processing.

What is working:

  • I can successfully analyze an audio track and create a 2D waveform texture. (With some hacks using Sound object, more on that below)
  • I can set up an event instance to further manipulate the audio.

What is NOT working:

  • I can only create this 2D waveform by creating a separate Sound object using createSound(filepath) and the filepath of that audio track. Meaning it is not being created from the event. This is cumbersome and not ideal as it involves keeping track of a regular audio file path and a FMOD Studio event.

  • I can instead use an FMOD event, but can’t create a “static” 2D waveform, as the only way I have been able to visualize the event is by displaying the spectrum data on each frame update.


This is ideally what I want to visualize, but like I said can only do this with a created Sound object.

Here are the relevant parts of my main script:

private void Start()
    {
        // Initialize the sprite renderer.
        sprend = this.GetComponent<SpriteRenderer>();

        //Prepare FMOD event, sets _event.
        PrepareFMODEventInstance();

        _samples = new float[_windowSize];

        // Get the waveform and add it to the sprite renderer.
        Texture2D texwav = GetWaveformFMOD();
        Rect rect = new Rect(Vector2.zero, new Vector2(width, height));
        sprend.sprite = Sprite.Create(texwav, rect, Vector2.zero);

       // Rest of start ...
    }

 private Texture2D GetWaveformFMOD()
    {
        int halfheight = height / 2;
        float heightscale = (float)halfheight * 0.0025f;

        // get the sound data
        Texture2D tex = new Texture2D(width, height, TextureFormat.RGBA32, false);
        waveform = new float[width];

        // THIS IS WHERE IM LIMITED BY USING AN AUDIO FILE PATH.
        samples = GetSampleData("Assets/music 1.mp3");
        samplesize = samples.Length;

        // Debug log to check if the is valid and has data
        UnityEngine.Debug.Log("Samples: " + samples);

        int packsize = (samplesize / width);
        for (int w = 0; w < width; w++)
        {
            waveform[w] = Mathf.Abs(samples[w * packsize]);
        }

        // Debug log to check the dimensions and content of the waveform array
        UnityEngine.Debug.Log("Waveform array length: " + waveform.Length);

        // map the sound data to texture
        // 1 - clear
        for (int x = 0; x < width; x++)
        {
            for (int y = 0; y < height; y++)
            {
                tex.SetPixel(x, y, background);
            }
        }

        // 2 - plot
        for (int x = 0; x < width; x++)
        {
            for (int y = 0; y < waveform[x] * heightscale; y++)
            {
                tex.SetPixel(x, halfheight + y, foreground);
                tex.SetPixel(x, halfheight - y, foreground);
            }
        }

        tex.Apply();

        // Debug log to check if the texture is being created and modified as expected
        UnityEngine.Debug.Log("Waveform texture created: " + tex.width + "x" + tex.height);

        return tex;

    }

private byte[] GetSampleData(string filePath)
    {
        // Very useful tool for debugging FMOD function calls
        FMOD.RESULT result;

        // Sound variable to retrieve the sample data from
        FMOD.Sound sound;

        // Creating the sound using the file path of the audio source 
        // Make sure to create the sound using the MODE.CREATESAMEPLE | MDOE.OPENONLY so the sample data can be retrieved
        result = FMODUnity.RuntimeManager.CoreSystem.createSound(filePath, FMOD.MODE.CREATESAMPLE | FMOD.MODE.OPENONLY, out sound);

        // Debug the results of the FMOD function call to make sure it got called properly
        if (result != FMOD.RESULT.OK)
        {
            UnityEngine.Debug.Log("Failed to create sound with the result of: " + result);
            return null;
        }

        // Retrieving the length of the sound in milliseconds to size the arrays correctly
        result = sound.getLength(out uint length, FMOD.TIMEUNIT.MS);

        if (result != FMOD.RESULT.OK)
        {
            UnityEngine.Debug.Log("Failed to retrieve the length of the sound with result: " + result);
            return null;
        }
    
        // Creating the return array which will have the sample data is a readable variable type 
        // Using the length of the sound to create it to the right size 
        byte[] byteArray = new byte[(int)length];

        // Retrieving the sample data to the pointer using the full length of the sound 
        result = sound.readData(byteArray);

        if (result != FMOD.RESULT.OK)
        {
            UnityEngine.Debug.Log("Failed to retrieve data from sound: " + result);
            return null;
        }

        UnityEngine.Debug.Log("Returning byte array of samples, result: " + result);

        //Returning the array populated with the sample data to be used
        return byteArray;
    }

// Prepare FMOD Event function for added context.
private void PrepareFMODEventInstance()
    {
        // Create the event instance from the event path, add 3D sound and start.
        _event = FMODUnity.RuntimeManager.CreateInstance(_eventPath);
        _event.set3DAttributes(FMODUnity.RuntimeUtils.To3DAttributes(gameObject.transform));
        //_event.start();

        // Create the FFT dsp, set window type, and window size.
        FMODUnity.RuntimeManager.CoreSystem.createDSPByType(FMOD.DSP_TYPE.FFT, out _dsp);
        _dsp.setParameterInt((int)FMOD.DSP_FFT.WINDOWTYPE, (int)_windowShape);
        _dsp.setParameterInt((int)FMOD.DSP_FFT.WINDOWSIZE, _windowSize * 2);

        // Get the channel group from the event and add to DSP.
        _event.getChannelGroup(out _channelGroup);
        _channelGroup.addDSP(0, _dsp);
    }

I feel like I am close to what I need, but I am a bit confused and not as knowledgeable about DSP and fmod in general. Am I just doing this all wrong and can a 2D “static” waveform be simply created using FFT spectrum data?

Thanks again for any help.

Hi,

There are a few options to do this:

  1. Retrieve the sound from the event instance
  2. Use the DSP_READ_CALLBACK to get the sound data. We have an example for Unity with a very similar functionality here: Unity Integration | Scripting Example - DSP Capture.

Let me know which works best for you and I am happy to elaborate.

Hope this helps!

I checked out DSP Capture script example before making this post, but I didn’t really understand what was going on.

How can I retrieve the sound from the event instance? The only way I found was to give the audio file path and like I said I would rather handle everything from the event instance itself.

1 Like

Hi,

To get the sound from an event instance you can:

  1. Get the channel group from the event instance after it is playing:
if (eventInstance.hasHandle() && !sound.hasHandle())
{
	if (eventInstance.getPlaybackState(out FMOD.Studio.PLAYBACK_STATE state) == FMOD.RESULT.OK)
	{
		if (state == FMOD.Studio.PLAYBACK_STATE.PLAYING)
		{
			if (eventInstance.getChannelGroup(out FMOD.ChannelGroup group) == FMOD.RESULT.OK)
			{
				sound = FindCurrentSound(group);
				sound.getName(out string name, 256);
				Debug.Log($"Audio Name: {name}");
			}
		}
	}
}
  1. Then I wrote a function to find the sound from the channel group
FMOD.Sound FindCurrentSound(FMOD.ChannelGroup channelGroup)
{
	FMOD.Sound sound = new FMOD.Sound();

	// Check if we are currently in the right channel group
	if (channelGroup.getNumChannels(out int numChannels) == FMOD.RESULT.OK && numChannels > 0)
	{
		if (numChannels > 1)
		{
			Debug.Log("More than 1 channel");
		}
		else
		{
			if (channelGroup.getChannel(0, out FMOD.Channel channel) == FMOD.RESULT.OK)
			{
				if (channel.getCurrentSound(out sound) == FMOD.RESULT.OK)
				{
					Debug.Log("Found the correct Sound");
				}
				else
				{
					Debug.Log("Could not retrieve sound");
					sound = new FMOD.Sound();
				}
			}
		}
	}

	// Check if the sound has been found
	if (!sound.hasHandle())
	{
		// If not, continue the recusion
		channelGroup.getName(out string name, 256);
		Debug.Log(name);

		if (channelGroup.getNumGroups(out int numGroups) == FMOD.RESULT.OK && numGroups > 0)
		{
			if (numGroups > 1)
			{
				Debug.Log("More than one Group");
			}
			else
			{
				if (channelGroup.getGroup(0, out FMOD.ChannelGroup child) == FMOD.RESULT.OK)
				{
					sound = FindCurrentSound(child);
				}
			}
		}
	}
	// Else the sound has been found so return the sound


	// This will either be a real sound or not depending if it was found in the recussion.
	return sound;
}

This only works for events with a single audio track.

Hope this helps!

Hey there, I tried your above solution with my code for drawing the waveform:
It did not work however, giving me the error:

Failed to retrieve the length of the sound with result: ERR_INVALID_PARAM
UnityEngine.Debug:Log (object)
TrackFMOD:GetSampleData (FMOD.Sound) (at Assets/TrackFMOD.cs:248)
TrackFMOD:GetWaveformFMOD () (at Assets/TrackFMOD.cs:175)
TrackFMOD:Start () (at Assets/TrackFMOD.cs:51)

NullReferenceException: Object reference not set to an instance of an object
TrackFMOD.GetWaveformFMOD () (at Assets/TrackFMOD.cs:176)
TrackFMOD.Start () (at Assets/TrackFMOD.cs:51)

/**
 * Track FMOD script - visualizes an FMOD Event and creates a "Track" component for playback. 
 **/

using UnityEngine;
using System;
using System.Collections;
using System.Runtime.InteropServices;

public class TrackFMOD : MonoBehaviour
{
    public int width = 1024;
    public int height = 256;
    public Color background = Color.black;
    public Color foreground = Color.yellow;
    
    public GameObject arrow = null;
    public Camera cam = null;

    ///////////// FMOD DSP stuff
    public FMODUnity.EventReference _sampleInstancePath;
    public int _windowSize = 512;
    public FMOD.DSP_FFT_WINDOW _windowShape = FMOD.DSP_FFT_WINDOW.RECT;

    private FMOD.Studio.EventInstance _sampleInstance;
    private FMOD.Channel _channel;
    private FMOD.ChannelGroup _channelGroup;
    private FMOD.DSP _dsp;
    private FMOD.DSP_PARAMETER_FFT _fftparam;

    ///////////// FMOD sound
    private FMOD.Sound _sound;

    // Private sprite and waveform fields.
    private SpriteRenderer sprend = null;
    private int samplesize;
    private byte[] _samples = null;
    private float[] waveform = null;
    private float arrowoffsetx;


    private void Start()
    {
        // Initialize the sprite renderer.
        sprend = this.GetComponent<SpriteRenderer>();

        // Prepare FMOD event, sets _sampleInstance.
        PrepareFMODEventInstance();

        // Get the waveform and add it to the sprite renderer.
        Texture2D texwav = GetWaveformFMOD();
        Rect rect = new Rect(Vector2.zero, new Vector2(width, height));
        sprend.sprite = Sprite.Create(texwav, rect, Vector2.zero);

        _sound.setMode(FMOD.MODE.LOOP_NORMAL);


        // Adjust arrow and waveform sprite.
        arrow.transform.position = new Vector3(0f, 0f, 1f);
        arrow.transform.Translate(Vector3.left * (sprend.size.x / 2f));
        arrowoffsetx = -(arrow.GetComponent<SpriteRenderer>().size.x / 2f);

        sprend.transform.Translate(Vector3.left * (sprend.size.x / 2f));

    }

    private void Update()
    {

    }

    // Prepare FMOD Event Instance.
    private void PrepareFMODEventInstance()
    {
        // Create the event instance from the event path, add 3D sound and start.
        _sampleInstance = FMODUnity.RuntimeManager.CreateInstance(_sampleInstancePath);
        _sampleInstance.set3DAttributes(FMODUnity.RuntimeUtils.To3DAttributes(gameObject.transform));
        _sampleInstance.start();

        // Create the FFT dsp, set window type, and window size.
        FMODUnity.RuntimeManager.CoreSystem.createDSPByType(FMOD.DSP_TYPE.FFT, out _dsp);
        _dsp.setParameterInt((int)FMOD.DSP_FFT.WINDOWTYPE, (int)_windowShape);
        _dsp.setParameterInt((int)FMOD.DSP_FFT.WINDOWSIZE, _windowSize * 2);

        // Get the channel group from the event and add to DSP.
        _sampleInstance.getChannelGroup(out _channelGroup);
        _channelGroup.addDSP(0, _dsp);

        // test
        if (_sampleInstance.hasHandle() && !_sound.hasHandle())
        {
            if (_sampleInstance.getPlaybackState(out FMOD.Studio.PLAYBACK_STATE state) == FMOD.RESULT.OK)
            {
                if (state == FMOD.Studio.PLAYBACK_STATE.PLAYING)
                {
                    if (_sampleInstance.getChannelGroup(out FMOD.ChannelGroup group) == FMOD.RESULT.OK)
                    {
                        _sound = FindCurrentSound(group);
                        _sound.getName(out string name, 256);
                        Debug.Log($"Audio Name: {name}");
                    }
                }
            }
        }
    }

    FMOD.Sound FindCurrentSound(FMOD.ChannelGroup channelGroup)
    {
        FMOD.Sound sound = new FMOD.Sound();

        // Check if we are currently in the right channel group
        if (channelGroup.getNumChannels(out int numChannels) == FMOD.RESULT.OK && numChannels > 0)
        {
            if (numChannels > 1)
            {
                Debug.Log("More than 1 channel");
            }
            else
            {
                if (channelGroup.getChannel(0, out FMOD.Channel channel) == FMOD.RESULT.OK)
                {
                    if (channel.getCurrentSound(out _sound) == FMOD.RESULT.OK)
                    {
                        Debug.Log("Found the correct Sound");
                    }
                    else
                    {
                        Debug.Log("Could not retrieve sound");
                        sound = new FMOD.Sound();
                    }
                }
            }
        }

        // Check if the sound has been found
        if (!sound.hasHandle())
        {
            // If not, continue the recusion
            channelGroup.getName(out string name, 256);
            Debug.Log(name);

            if (channelGroup.getNumGroups(out int numGroups) == FMOD.RESULT.OK && numGroups > 0)
            {
                if (numGroups > 1)
                {
                    Debug.Log("More than one Group");
                }
                else
                {
                    if (channelGroup.getGroup(0, out FMOD.ChannelGroup child) == FMOD.RESULT.OK)
                    {
                        sound = FindCurrentSound(child);
                    }
                }
            }
        }
        // Else the sound has been found so return the sound

        Debug.Log("Sound found...\n");
        // This will either be a real sound or not depending if it was found in the recussion.
        return sound;
    }


    private Texture2D GetWaveformFMOD()
    {
        int halfheight = height / 2;
        float heightscale = (float)halfheight * 0.0025f;

        // get the sound data
        Texture2D tex = new Texture2D(width, height, TextureFormat.RGBA32, false);
        waveform = new float[width];

        // get samples from the helper function.
        _samples = GetSampleData(_sound);
        samplesize = _samples.Length;


        // Debug log to check if the is valid and has data
        UnityEngine.Debug.Log("Samples: " + _samples);

        int packsize = (samplesize / width);
        for (int w = 0; w < width; w++)
        {
            waveform[w] = Mathf.Abs(_samples[w * packsize]);
        }

        // Debug log to check the dimensions and content of the waveform array
        UnityEngine.Debug.Log("Waveform array length: " + waveform.Length);

        // map the sound data to texture
        // 1 - clear
        for (int x = 0; x < width; x++)
        {
            for (int y = 0; y < height; y++)
            {
                tex.SetPixel(x, y, background);
            }
        }

        // 2 - plot
        for (int x = 0; x < width; x++)
        {
            for (int y = 0; y < waveform[x] * heightscale; y++)
            {
                tex.SetPixel(x, halfheight + y, foreground);
                tex.SetPixel(x, halfheight - y, foreground);
            }
        }

        tex.Apply();

        // Debug log to check if the texture is being created and modified as expected
        UnityEngine.Debug.Log("Waveform texture created: " + tex.width + "x" + tex.height);

        return tex;

    }


    /// <summary>
    /// Return sample data in a byte array from an audio source using its file path 
    /// </summary>
    /// <param name="filePath"></param>
    /// <returns></returns>
    private byte[] GetSampleData(FMOD.Sound sound)
    {
        // Very useful tool for debugging FMOD function calls
        FMOD.RESULT result;

        // Creating the sound using the file path of the audio source 
        // Make sure to create the sound using the MODE.CREATESAMEPLE | MDOE.OPENONLY so the sample data can be retrieved
        // result = FMODUnity.RuntimeManager.CoreSystem.createSound(filePath, FMOD.MODE.CREATESAMPLE | FMOD.MODE.OPENONLY, out _sound);

        // Debug the results of the FMOD function call to make sure it got called properly
        //if (result != FMOD.RESULT.OK)
        //{
        //    UnityEngine.Debug.Log("Failed to create sound with the result of: " + result);
        //    return null;
        //}

        // Retrieving the length of the sound in milliseconds to size the arrays correctly
        result = _sound.getLength(out uint length, FMOD.TIMEUNIT.MS);
   

        if (result != FMOD.RESULT.OK)
        {
            UnityEngine.Debug.Log("Failed to retrieve the length of the sound with result: " + result);
            return null;
        }
    
        // Creating the return array which will have the sample data is a readable variable type 
        // Using the length of the sound to create it to the right size 
        byte[] byteArray = new byte[(int)length];

        // Retrieving the sample data to the pointer using the full length of the sound 
        result = _sound.readData(byteArray);

        if (result != FMOD.RESULT.OK)
        {
            UnityEngine.Debug.Log("Failed to retrieve data from sound: " + result);
            return null;
        }

        UnityEngine.Debug.Log("Returning byte array of samples, result: " + result);

        //Returning the array populated with the sample data to be used
        return byteArray;
    }


}

The line it failed at was in GetSampleData:

// Retrieving the length of the sound in milliseconds to size the arrays correctly
result = _sound.getLength(out uint length, FMOD.TIMEUNIT.MS);

I appreciate your help, however I think I am overcomplicating myself and I will try simply drawing a approximation of a waveform that is consistent with the event timeline’s length and simply represents that sound and allows playback controls.

I don’t think I need an actual waveform to be drawn just for the user to understand that sprite represents the length of that event and whatnot.

I will keep trying and update you all in the future.

1 Like

Hi,

I believe the error is being called because the sound is never retrieved since we never make it to FindCurrentChannel().

I understand, if you decide to return to this I am happy to assist where I can.

Do you mean FindCurrentSound(group)?

There is no FindCurrentChannel(). However I also noticed with this approach the event has to be playing for us to try and get the sound and then draw the waveform.

Ideally I would want it to analyze the event and create the waveform2d sprite before playback is started.

Correct.

An option may be to start the event and immediately pause it, allowing you to find the sound.

Hi! Were you ever able to achieve this? I’m currently on a very similar situation as you and nothing seems to be working haha.

Hi,

What version of the FMOD integration are you using? Where are you getting stuck?