General advice on user-friendly process for players to upload audio assets then pipe them through FMOD-powered Unity?

I’m looking for Unity-specific ways of handling this, and any limitations that may be part of Unity.

Say there’s a game with UGC (User-Generated Content), where you can record and even upload sounds from local storage into the game, to play for others. For example, you have a sound of your cat meowing, and you import it into the virtual world so that other players can hear it.

In a case like this, the sound originates outside of an FMOD bank, isn’t loaded in at runtime, but FMOD is used to process and effect it — whether that’s spatialization or adding reverb.

How might this be handled?

Hi,

An option you have is using Programmer Instrument, explained in the Glossary. These allow for files outside of the FMOD and game project files to be used as audio sources.

You will need to include: using System; and using System.Runtime.InteropServices; at the top of your code file.

Create an FMODUnity.EventReference variable which will be used to instantiate your Programmer Instrument from the FMOD Project.

Then create an FMOD.Studio.EVENT_CALLBACK variable which will be used to create the sound to be played by FMOD.

In

void Start()
{
	yourPlayBackVariable = new FMOD.Studio.EVENT_CALLBACK(PlayFileCallBack);
}

This instantiates your EVENT_CALLBACK to be used in the PlayAudioFile(string location) function. Call this when the sound is ready to be played, how it gets the parameter location is up to you. Be sure that location is the full file path to the audio source.

PlayAudioFile(string location) Function
public void PlayAudioFile(string location)
{
	var eventInstance = FMODUnity.RuntimeManager.CreateInstance(yourEventReference);
	
	// FMOD.RESULT is a very useful debugging tool to ensure that FMOD functions have run properly 
	FMOD.RESULT result; 
	if (yourEventInstance.isValid() == false)
	{
		Debug.Log("Failed to create instance, instance invalid");
		return;
	}
	// Lock the file location in memory to be used in the CALLBACK function to find the audio source
	GCHandl stringLocation = GCHandle.Alloc(location);
	// Assigning this data to the created instance 
	result = eventInstnace.setUserData(GCHandle.ToIntPtr(stringLocation));
	
	// Using an if statement like this after most FMOD function calls will help debug errors in code 
	if (result != FMOD.RESULT.OK)
	{
		Debug.Log("Failed to create evenInstance with a result of " + result);
		return;
	}
	
	
	// Check the result after each function call 
	result = eventInstance.setCallback(yourPlayBackVaraible); 
	result = eventInstance.start();
	// The sound has successfully been created and played 
}

This is your FMOD.Studio.EVENT_CALLBACK function used to create the sound for FMOD to play using the audio source file location.

CALLBACK Function
[AOT.MonoPInvokeCallback(typeof(FMOD.Studio.EVENT_CALLBACK))]
    static FMOD.RESULT YourCallBackFunction(FMOD.Studio.EVENT_CALLBACK_TYPE type, IntPtr instancePtr, IntPtr parameterPtr)
    {
        FMOD.Studio.EventInstance instance = new FMOD.Studio.EventInstance(instancePtr);

        if (instance.isValid() == false)
        {
            UnityEngine.Debug.Log("Failed to create instance, instance invalid");
            return FMOD.RESULT.ERR_EVENT_NOTFOUND;
        }

        //Retrive the user data 
        IntPtr stringPtr;
        instance.getUserData(out stringPtr);

        // Retrieving the path of the audio source from the user data
        GCHandle stringHandle = GCHandle.FromIntPtr(stringPtr);
        String location = stringHandle.Target as String;

        switch (type)
        {
            case FMOD.Studio.EVENT_CALLBACK_TYPE.CREATE_PROGRAMMER_SOUND:
                {
                    FMOD.MODE soundMode = FMOD.MODE.LOOP_NORMAL | FMOD.MODE.CREATECOMPRESSEDSAMPLE | FMOD.MODE.NONBLOCKING;
                    var parameter = (FMOD.Studio.PROGRAMMER_SOUND_PROPERTIES)Marshal.PtrToStructure(parameterPtr, typeof(FMOD.Studio.PROGRAMMER_SOUND_PROPERTIES));
		
                    FMOD.Sound audioSound;
                    FMOD.RESULT result;

                    // Creating the sound using the location of the audio source 
                    result = FMODUnity.RuntimeManager.CoreSystem.createSound(location, soundMode, out audioSound);

                    if (result == FMOD.RESULT.OK)
                    {
                        parameter.sound = audioSound.handle;
                        parameter.subsoundIndex = -1;
                        Marshal.StructureToPtr(parameter, parameterPtr, false);
                    }
                    break;
                }
            case FMOD.Studio.EVENT_CALLBACK_TYPE.DESTROY_PROGRAMMER_SOUND:
                {
                    var paramerter = (FMOD.Studio.PROGRAMMER_SOUND_PROPERTIES)Marshal.PtrToStructure(parameterPtr, typeof(FMOD.Studio.PROGRAMMER_SOUND_PROPERTIES));
                    var sound = new FMOD.Sound(paramerter.sound);
                    sound.release();

                    break;
                }
            case FMOD.Studio.EVENT_CALLBACK_TYPE.DESTROYED:
                {
                    // Now the event has been destroyed, unpin the string memory so it can be garbaged collected
                    stringHandle.Free();
                    break;
                }
        }
        return FMOD.RESULT.OK;
    }

Once this is all set up users will be able to pass in the location of audio sources which you can then play using FMOD.

A full script example of Programmer Sounds being used can be found under Scripting Examples | Programmer Sounds. Keep in mind, in the example it is using an audio table and not the file location of an audio source, but it may be a useful reference.

Hope this helps!

3 Likes

@Connor_FMOD You are full of solutions! Thanks for laying this out.

I’ve been reading " 14.3 Loading DLC and UGC Banks", and… are there general pros/cons between these two approaches? The two being:

(1) Programmer Instruments
vs.
(2) Loading UGC banks

It seems like having users do a “UGC Bank” would require more legwork and familiarity with FMOD, but potentially access to other adaptive parameters.

Whereas using Programmer Instruments would restrict the the signal flow to what we (as the game company) dictate in the FMOD Studio Project, but the tradeoff is it makes things easier for audio newcomers who upload audio into our system.

Are there any other considerations to be aware of when deciding on an approach?

@Connor_FMOD And to add another related question about this and Programmer Instruments in particular, how is this use case supported by FMOD?

  • Unity - Manual: Microphone is used to live record a player speaking.
  • We pass this along internally and store the audio file on a server. At which point it’s out of Unity’s hands.
  • FMOD calls this audio file from the server and then it’s piped through an event with a Programmer Instrument.

Are there any known subtleties about how this works? Especially because it’s highly recommended that Unity built-in audio (aka old FMOD Core) is disabled ENTIRELY… but I presume Microphone class exists separately of that, so that functionality would still exist and there is no codependency?

In other words: FMOD does not conflict with Unity audio input?

UPDATE to this is that we found that disabling Unity soundsystem also disables audio input via Microphone (!), so we need to find another way around. Is there a more straightforward guide to achieve that use case?

I’ve been googling and found this video which is one of the closer things I’ve found for start-to-finish documentation: FMOD & Unity | Recording The Players Voice And Playing It Back At Runtime - YouTube

Are there any other resources we should be aware of? Whether provided by FMOD or user-created.

Hi,

There isn’t much more to take into consideration other than what you have pointed out. Which method you end up using will solely depend on the functionality you want to give the user.

Scott’s video is a great resource to get started with, but there are some changes we will want to make ourselves.
• Includes required

using FMODUnity;
using UnityEngine;
using System.Collections;
using System.Runtime.InteropServices;
using System.Collections.Generic;
using System.IO;

• Extra functions used

ERRCHECK()
/// <summary>
/// Use to debug FMOD functions with explanation of what the function was doing with an both success and failed messages 
/// </summary>
/// <param name="result"></param>
/// <param name="failed"></param>
/// <param name="success"></param>
private void ERRCHEK(FMOD.RESULT result, string failed, string success)
{
#if UNITY_EDITOR
    if (result.ToString().Length != 0)
	{
		if (result != FMOD.RESULT.OK)
			Debug.Log(failed + " " + result);
		else
			Debug.Log(success + " " + result);
	}
#endif
}

/// <summary>
/// Override of no success message is required 
/// </summary>
/// <param name="result"></param>
/// <param name="failed"></param>
private void ERRCHEK(FMOD.RESULT result, string failed)
{
	//Will only call debugs if in editor to save on calls in build 
#if UNITY_EDITOR
	if (result.ToString().Length != 0)
	{
		if (result != FMOD.RESULT.OK)
			Debug.Log(failed + " " + result);
	}
#endif
}
SetState()
/// <summary>
/// Sets the state of the recording
/// </summary>
/// <param name="currentState">Options: Recording, Stopped, Restart</param>
private void SetState(string currentState)
{
	switch(currentState)
    {
		case "Recording":
            {
				recording = true;
				stopped = false;
				killed = false;
#if UNITY_EDITOR
				Debug.Log("RECORDING");
#endif
				break;
            }
		case "Stopped":
			{
				recording = false;
				stopped = true;
				killed = true;
				soundActive = false;
#if UNITY_EDITOR
				Debug.Log("STOPPED");
#endif
				break;
			}
		case "Restart":
			{
				recording = false;
				stopped = false;
				killed = false;
				soundActive = true; 
#if UNITY_EDITOR
				Debug.Log("RESTARTING");

#endif
				break;
			}
	}
}

• You will need two systems Core System, one to record and one to write to a .wav file

Creating Systems
//System Variables
private FMOD.System micRecordSystem, wavWriterSystem;
private string micRecordData = "MICTEST";
private string wavWiterData = "WAVWRITER";

//Creating systems to record and output mic input -------------------------------------------------------------
GCHandle micRecordDataPtr = GCHandle.Alloc(micRecordData);
GCHandle wavWriterDataPrt = GCHandle.Alloc(wavWiterData);

FMOD.RESULT result;
FMOD.OUTPUTTYPE currentOutput;
//Creating the mic system solely for recording mic input 
result = FMOD.Factory.System_Create(out micRecordSystem);
ERRCHEK(result, "Failed to create mic sysytem"); 
//Initialize mic record system 
resusystemcRecordSystem.init(100, FMOD.INITFLAGS.NORMAL, GCHandle.ToIntPtr(micRecordDataPtr));
ERRCHEK(result, "Mic system int failed");
//Checking current output for the system 
result = micRecordSystem.getOutput(out currentOutput);
ERRCHEK(result, "Failed to check output", "Mic system current ouput " + currentOutput.ToString() + " with result"); 

FMOD.RESULT wavResult; //Just for this snip
//Creating the wav writer system 
wavResult= FMOD.Factory.System_Create(out wavWriterSystem);
ERRCHEK(wavResult, "Failed to create wav system"); 
//Initialize the wav writer system 
wavResult= wavWriterSystem.init(100, FMOD.INITFLAGS.NORMAL, GCHandle.ToIntPtr(wavWriterDataPrt));
ERRCHEK(wavResult, "Failed to init wav write system"); 
		
//Assigning output of Wav system to nosound, till we are ready to record sound
wavResult= wavWriterSystem.setOutput(FMOD.OUTPUTTYPE.NOSOUND);
wavWriterSystem.getOutput(out currentOutput); 
ERRCHEK(wavResult, "Failed to change wav output", "Wav system current output " + currentOutput.ToString() + " with result ");
//-------------------------------------------------------------------------------------------------------------

• Because we are creating these systems ourselves, we are responsible for updating and releasing them
• Call in Update()

Update systems
//Updating both the created systems
if (!killed)
{
	micRecordSystem.update();
	wavWriterSystem.update();
}
Release()
//REMEMBER TO RELEASE SYSTEMS SO THERE ARENT MEMORY LEAKS, this will get called if you stop the Editor running or if the object is destroyed
private void OnDestroy()
{ 
	micRecordSystem.release();
	wavWriterSystem.release();
}

• We can follow Scott’s example of how to set up and retrieve the recordingDevice from our new system
• When creating the new sound don’t use the FMODUnity.RuntimeManger.CoreSystem you will want to you’re your recordingSystem
• When beginning the recording, input will be recorded on the recordingSystem, while the wavWriterSystem will only be responsible for outputting the file

Start Recording
//START RECORDING----------------------------------------------------------------------------------------------
if (Input.GetKeyDown(KeyCode.R) && !recording && soundActive)
{
	SetState("Recording");

	FMOD.RESULT result;

	result = micRecordSystem.recordStart(recordingDeviceIndex, sound, true);
	ERRCHEK(result, "Failed to start recording"); 

	//Playing the sound to the mic record system 
	result = micRecordSystem.playSound(sound, channelGroup, false, out channel);
	ERRCHEK(result, "Failed to play sound on mic system");

	//Playing the sound on the wav system 
	result = wavWriterSystem.playSound(sound, channelGroup, false, out wavChannel);
	ERRCHEK(result, "Failed to play sound on wav system");
			
	//Start wav system writing to file, make sure this only happens once 
	result = wavWriterSystem.setOutput(FMOD.OUTPUTTYPE.WAVWRITER);
	ERRCHEK(result, "Wav system failed to start writing to file", "Wav system now writing to file");
}
//-------------------------------------------------------------------------------------------------------------

• When the recording is done, we will need to stop writing to the .wav file so it can be used as a source for our Programmer Sound.

Stop Recording
//STOP RECORDING-----------------------------------------------------------------------------------------------
if (Input.GetKeyDown(KeyCode.S) && !stopped && recording)
{
	SetState("Stopped");

	//Stopping the recording
	FMOD.RESULT result = micRecordSystem.recordStop(recordingDeviceIndex);
	ERRCHEK(result, "Failed to stop recording on mic system");

	//Should stop WASAPI starving------------------------------------------------------------------------------
	result = wavChannel.stop();
	ERRCHEK(result, "Failed to stop wav channel");

	result = channel.stop();
	ERRCHEK(result, "Failed to stop mic channel"); 
	//---------------------------------------------------------------------------------------------------------
	result = sound.release();
	ERRCHEK(result, "Failed to release sound");

    //Stop the wav system writing to file 
    result = wavWriterSystem.setOutput(FMOD.OUTPUTTYPE.NOSOUND);
    ERRCHEK(result, "Failed to change wav system output");
}
//-------------------------------------------------------------------------------------------------------------

• If you want to do another recording, you will have to recreate the sound and delete the .wav file so it can’t be created again.

Recreate Sound
//RECREATE SOUND-----------------------------------------------------------------------------------------------
if (Input.GetKeyDown(KeyCode.L) && killed && stopped)
{
	SetState("Restart"); 

	//Only need to recreate the sound 
	FMOD.RESULT result = micRecordSystem.createSound("Sound", FMOD.MODE.LOOP_NORMAL | FMOD.MODE.OPENUSER,
	ref exinfo, out sound);
	ERRCHEK(result, "Failed to recreate sound");

	//Removing old wav so that the new one can be recorded 
	if (filePath.text.Length != 0)
	{
		File.Delete(filePath.text);
#if UNITY_EDITOR
		Debug.Log("Deleted .wav file");
#endif
	}
}
//-------------------------------------------------------------------------------------------------------------

I’ll include the entire file. But it’s a good idea to try to set it up yourself so you understand how it works!
RecordMic.txt (9.4 KB)

I have also included how to use programmer sounds
ProgrammerSounds.txt (3.6 KB)

Hope this helps!

@Connor_FMOD Thank you for all the specific references! Appreciate your time with this. We’ll take a closer look.

1 Like