IOS - Sound plays through receiver when using voice chat

korimart2 · September 30, 2022, 9:14am

Hi

Our project requires enabling voice chat while playing sounds through fmod at the same time. However sound does not play as expected when AVAudioSession category and mode are configured for voice chat.

Reproduction steps are as follows.

Start playing any sound (UFMODAudioComponent::Play)
Call IOSAppDelegate::EnableVoiceChat and IOSAppDelegate::EnableHighQualityVoiceChat. (This is equivalent to calling AVAudioSession::setCategory with AVAudioSessionCategoryPlayAndRecord with mode AVAudioSessionModeVoiceChat with options AVAudioSessionCategoryOptionDefaultToSpeaker | AVAudioSessionCategoryOptionMixWithOthers)
Sound starts playing through both the speaker and the receiver. (If was connected to a bluetooth device, sound stops).

We expect that sound plays though only the speaker (not the reciever), and when connected to a bluetooth device, through that device only.

Everything is ok when voice chat is not enabled, but our voice chat module (Vivox) requires that we set AVAudioSessionCategory to those listed above.

Thanks

jeff_fmod · October 7, 2022, 4:53am

I have not been able to reproduce this issue. Can you please tell me what version of FMOD and Unreal you are using?

korimart2 · October 7, 2022, 12:19pm

We are using ue4 4.27.2 and FMOD 2.02.09.

After your reply I found out that the issue might have something to do with the IOS version. We have two iPhone 8 devices one with IOS version 15.5 and one with 15.6.1. I installed the same ipa file to both devices and the issue was reproduced on 15.6.1 and not on 15.5

Below is the code I used.

	TSoftObjectPtr<UFMODEvent> Event{ FSoftObjectPath{ TEXT("PATH") } };
	FMODAudio->SetEvent(Event.LoadSynchronous());
	FMODAudio->Play();

	[[IOSAppDelegate GetDelegate] EnableVoiceChat: YES];
	[[IOSAppDelegate GetDelegate] EnableHighQualityVoiceChat: YES];

Thanks again for your help

jeff_fmod · October 14, 2022, 12:36am

Thank you for the additional information. After some discussion with the development team I’m not sure there is much we can do about this issue unfortunately; we don’t have any control over which output device we send audio to on iOS as this is handled by AVAudioSessionCategory / AVAudioSessionCategoryOptions, which are expected to be set by the user depending on their unique requirements. If you are finding this issue does not reproduce with regular Unreal audio however then there may be an issue with FMOD that we can investigate further.
Does the bluetooth issue still reproduce on 15.5? I saw in the Vivox docs that you also need to set AVAudioSessionCategoryOptionAllowBluetooth and/or AVAudioSessionCategoryOptionAllowBluetoothA2DP to allow audio to send to a bluetooth device, which may be worth investigating if you haven’t yet.

korimart2 · October 20, 2022, 6:33am

Yes, the bluetooth issue is still reproduced and it might not have been related to IOS version after all. I tested with another iPhone 12 device with IOS 15.5 installed and it was no different than with 15.6.1. It might be that the device that looked normal happened to have its receiver broken. I’m sorry for the misinformation.

When tested with the default audio library, sound did play with a bluetooth device although it was quite laggy for some reason. Below is the code I used. (EAudioFeature::BluetoothMicrophone sets AVAudioSessionCategoryOptionAllowBluetooth)

	TSoftObjectPtr<USoundWave> Event{ FSoftObjectPath{ TEXT("PATH'") } };
	UGameplayStatics::PlaySound2D(this, Event.LoadSynchronous());

	[[IOSAppDelegate GetDelegate] SetFeature: EAudioFeature::BluetoothMicrophone Active: YES];
	[[IOSAppDelegate GetDelegate] EnableVoiceChat: YES];
	[[IOSAppDelegate GetDelegate] EnableHighQualityVoiceChat: YES];

Thanks

jeff_fmod · October 28, 2022, 4:45am

Apologies for the delayed response, and thank you for the additional information. Unfortunately I have still been unable to reproduce this issue; does removing the AVAudioSessionCategoryOptionDefaultToSpeaker flag help at all? That is the only one I can see in the AVAudioSession docs that might explain why it is being forced to the receiver and not the bluetooth headset.
Bluetooth does have significantly more latency than a wired connection. There are a few things you can do to try to reduce latency which you can find in the Platforms | iOS section of the docs.

korimart2 · November 9, 2022, 6:07am

Hi. I’m sorry for the delayed reply. We are still experiencing the issue even with the AVAudioSessionCategoryOptionDefaultToSpeaker suggestion. I’m re-listing the exact setup and reproduction steps here to avoid any confusion.

Mobile Device: iPhone 12 with IOS 15.5
Bluetooth Device: Samsung Galaxy Buds Pro
UE4 version: 4.27 built from source with minimal modification in IOSToolChain.cs to ignore unused variable errors with Xcode 13
FMOD version: 2.02.09 for UE4.27

Issue: Sound does not play with the following code. (Sound plays normally if IOSAppDelegate calls are removed)

	TSoftObjectPtr<UFMODEvent> Event{ FSoftObjectPath{ TEXT("FMODEvent'/Game/FMOD/Events/BGM_New/InGame/BGM_Game_Loading.BGM_Game_Loading'") } };
	FMODAudio->SetEvent(Event.LoadSynchronous());
	FMODAudio->Play();

	[[IOSAppDelegate GetDelegate] SetFeature: EAudioFeature::BluetoothMicrophone Active: YES];
	[[IOSAppDelegate GetDelegate] SetFeature: EAudioFeature::UseReceiver Active: YES];
	[[IOSAppDelegate GetDelegate] EnableVoiceChat: YES];
	[[IOSAppDelegate GetDelegate] EnableHighQualityVoiceChat: YES];

FMOD settings in the project are as defaulted.

I will send you a minimal project that demonstrates the issue (at least on my device) if it is okay with you.

Thanks again for your help.

jeff_fmod · November 16, 2022, 12:58am

A minimal reproduction would be appreciated as I still haven’t been able to reproduce this issue unfortunately. If you could please upload it to your FMOD Profile I will take a look.

NexonKorea · November 21, 2022, 5:11am

I uploaded a link to the project on this account’s profile. Please have a look and let me know if you have any issues building the project.

Thanks

jeff_fmod · November 25, 2022, 3:02am

Thank you for sending that over, I can reproduce the initial issue of audio outputting through the receiver, but I am finding bluetooth is behaving as expected on my test device.
Audio outputting through the receiver is expected behaviour when using the UseReceiver feature. On iOS the output device is set by IOSAppDelegate calls, as you have done here, and we do not have any additional control over which speaker to send audio to. If I remove the UseReceiver line from your repro or change it to NO, audio outputs through the speaker on iOS:

[[IOSAppDelegate GetDelegate] SetFeature: EAudioFeature::UseReceiver Active: NO];

I am also finding that FMOD is behaving exactly the same as Unreal’s built-in audio system, both in regards to outputting through the receiver and bluetooth device connection.

Can you please try removing the call to SetFeature: EAudioFeature::UseReceiver and see if that results in audio outputting from your device’s speaker instead of the receiver?

korimart2 · December 12, 2022, 10:40am

Thank you for taking a look at the repo.

The issues are still here even after setting NO to for SetFeature: EAudioFeature::UseReceiver, but I found out that everything is normal (almost) when tested with Airpods. It causes problems when the connected device is Samsung Galaxy Buds 2 or Buds Pro (which are the two devices I tested with but might be the same on other non-Apple devices).

I pushed a small change to the repo I shared with you and following the steps below, sound played either not at all, or through the phone (not the bluetooth device).

Open the app with no device connected. (sound should play normally here)
Call gg 1 1 through console. This is an Exec function I added.
Connect a bluetooth device (preferably a samsung device)
The bluetooth device plays the tone that’s usually played when a phone call ends and now sound starts playing through the phone.
If step 4 doesn’t happen, sound stops. In this case you might as well close the app and try again from step 1. I could observe the issue happening once every 2 tries.

Although Airpods did not stop playing sound with the above steps, with gg 1 1 the sound played like from a distance and was not so clear. It was all good with gg 0 0 and gg 1 0.

Thanks again for your time.

jeff_fmod · December 23, 2022, 5:31am

Apologies for the delayed response, unfortunately I still have not been able to reproduce this issue with your updates to the repo and using your exec function.
That said, I only have access to a generic bluetooth speaker and not Samsung Galaxy Buds or Airpods.
I will see if this is something we can procure and will test it again with a closer matching hardware setup.

korimart2 · January 5, 2023, 12:00pm

Hi. I have found some similar problems on Windows that might help in your investigation.

Vivox is required for the reproduction and you also need a bluetooth device that is displayed as “Hands-Free AG Audio” in Windows sound settings. In my case I tested with a Galaxy Buds Pro and a generic bluetooth headset from Beats by Dre. When the devices were first connected, on the sound panel, “Stereo” of the device was the default playback device and “Hands-Free AG Audio” was the default communication device and default recording device.

Everything is normal at first and sound plays nicely through FMOD but as soon as you enter a Vivox channel, all sound is muted. On the sound panel, the real time volume bar shows that the sound is playing through “Stereo” of the device but nothing is heard from the device itself.

Then if you disable “Stereo”, sound starts playing through “Hands-Free AG Audio” and you can actually hear it playing through the device. However, the sound is somewhat distorted and feels like from a distance (just like what happened on IOS). Still, both FMOD and Vivox do play. If you disable everything (both playback and recording) and turn “Stereo” back on, FMOD is normal but Vivox stops playing.

Since everything was normal before Vivox intervened, it might be that as Vivox starts using some OS functionality that involves Communication (Windows) or VoiceChat (iOS), devices that support playing this type of sound are causing issues trying to do their thing.

I am not familiar with any technical details so it might be something entirely different. My apolgoies if that is the case but I do hope that this is something relevant and can help you resolve the issues.

Also, please take a look at the second answer by OmaR1153 from this link windows 10 - What's the difference between stereo and hands free? - Super User

Might this be the cause of the issues? It might be possible that on iOS what happened was as the voice chat setting was switched back and forth the bluetooth devices were disabled and sound started playing through the main device which is the internal speaker of the phone.

Happy new year

jeff_fmod · January 13, 2023, 3:11am

Thank you for the additional information- we acquired some AirPods and Galaxy Buds but unfortunately we still cannot reproduce this issue:

Connecting headphones switches output to the headphones and sounds normal
Disconnecting headphones switches output to the Phone’s speakers
Reconnecting headphones switches output to the headphones and sounds normal

This could be related to communications ducking behavior on Windows. Can you please try the following:

Right click on the speaker icon in your toolbar
Select Sounds from the popup menu
Select the Communications tab
Set the When Windows detects communication activity option to “Do nothing”
Try disabling “Stereo” and see if both Vivox and FMOD play at a normal volume

Are you finding that this behavior only occurs with wireless devices, and is behaving as expected with single-audio-device hardware such as wired headphones?

korimart2 · January 16, 2023, 7:52am

Yes, things are normal with wired devices.

I have checked the setting and it was already at “Do nothing”.

I am still investigating as to why the issues might not be reproduced in your environment but in ours. I will leave a comment when I have more information. Since I’m close to being out of ideas, any suggestions would be greatly appreciated.

Thank you for the continued support.

jeff_fmod · January 18, 2023, 11:16pm

Thank you for confirming, I am thinking maybe FMOD and Vivox could compete for the audio device if Exclusive Mode is enabled. Can you please go to your device’s properties and ensure they have Exclusive Mode disabled?

On that properties page, can you please tell me what the Default Format for each of these devices are? If the hands-free device has a much lower sample rate that would explain why sound is coming out muffled. If FMOD is switching to a different device (which sounds like it might be the case) then there should be a log when it occurs. To confirm this, can you please enable Enable API Error Logging and set the Logging Level to “LEVEL LOG” and share the log you get while reproducing this issue?

When you say all sound is muted, is it all of your app’s sound or all of your system’s sound? For example, if you launch another application and play audio, does that play sound or is that muted as well?

korimart2 · January 27, 2023, 6:06am

This solved the problem of Vivox not playing and now both FMOD and Vivox work great, but only when in Stereo.

The hands-free device had a lower sample rate than Stereo. I wanted to see if I could make it better but the drop down was fixed and I could not change the sample rate. Thank you for this info.

I apologize for the reply being delayed. I got hit by covid and have been working from home. I will have the log you suggested and the answer to your last question when I get back to the office next week.

Thanks for being patient with us.

korimart2 · February 1, 2023, 7:20am

I tried the following steps again:

but for some reason, I could not reproduce the sound playing through the phone. But sound not playing persisted. Here is what I did:

Open the app with a bluetooth device connected.
Call gg 1 1.
Sound stops.

(Just for clarity, I did this on IOS)

PlzWork.log (111.1 KB)

Attached here is the log I obtained from the steps above.

I tried playing Apple Podcast and running the sample app. At first both Podcast and FMOD were okay but as soon as I typed gg 1 1 all sound (including the Podcast) stopped. When I checked the status bar Podcast had been paused and resuming it worked but once I returned to the sample app Podcast paused. It looked like the sample app was preventing both FMOD and Podcast from playing sound.

jeff_fmod · February 7, 2023, 5:49am

Thank you for the additional information- I hope you are feeling better now.
I have reproduced this issue now with the AirPods, I think the exec function isn’t working on my end, and was appearing as a false positive when I was testing previously.
Modifying the BeginPlay method to call gg(1, 1) or gg(1, 0), instead of using the Exec fucntion, is reproducing the behaviour you have described:

Audio plays quietly out of receiver
Connecting AirPods results in no audio coming out of the test app
Audio plays on other applications without issue
Disconnecting AirPods resumes audio playing out of receiver

Modifying the BeginPlay method to call gg(0, 0) or gg(0, 1) is behaving as expected:

Audio plays out of the device’s speakers
Connecting AirPods switches output to AirPods
Audio plays on other applications without issue
Disconnecting AirPods resumes audio playing out of device’s speakers

I have found that Unreal Audio plays back on AirPods when running with gg(1, 1), though the audio is crackling and distorted. In any case, I would expect FMOD’s behavior to match Unreal’s behavior, so I have passed this onto the dev team to investigate further- thank you for the reproduction and for your patience with helping me reproduce this issue.

korimart2 · March 3, 2023, 4:25am

Hello,

Is there any update on the investigation?