[Bug][2.02.06] Android startup crash

FMOD Engine
41

Hi @jeff_fmod , here are some clarifications:

  • We were on an old FMOD version previously, I forget which one, but the crash was not present.
  • We tried upgrading to the 2.02.12, which we put into production, and started getting the java.lang.UnsatisfiedLinkError: No implementation found for void org.fmod.FMOD.OutputAAudioHeadphonesChanged()
  • We reverted back to the previous version we were on and the crash was gone once again.
  • Later, we were required by other factors to upgrade the NDK version our app uses. We upgraded it to the latest version, and this required us to also upgrade FMOD once again (the app would not compile otherwise). We upgraded the NDK to version 25.1.8937393
  • We decided not to upgrade to the latest FMOD (2.02.13) because we knew that version caused this crash. We tried using 2.02.05 instead, thinking this thread indicated the crash was introduced in 2.02.06.
  • The crash is happening in production still and affects many of our users.

I can provide other details if needed, thanks

EDIT: Actually, I just noticed that the version we were using in production was still 2.02.12. So I will try truly downgrading to 2.02.05 which will likely solve the problem, but I am hoping we can update to later versions soon, but we cannot until this bug is fixed.

42

Thank you for the additional information, and apologies for the delayed response.
That error means the libfmod.so/libfmodL.so library has not been loaded. Some possibilities I can think of:

  • The libraries are being loaded too late. You can eliminate this possibility by loading libraries at static initialization time. We use this to load libraries in our shipped Android examples:
    public class MainActivity extends Activity implements OnTouchListener, Runnable
{
...
    static
    {
        for (String lib: BuildConfig.FMOD_LIBS)
        {
            System.loadLibrary(lib);
        }
        System.loadLibrary("example");
    }
}
  • The application was installed through something other than Google Play, and the libraries were not extracted to a standard location. To verify if this is a possibility, you could use Context.getPackageManager().getInstallSourceInfo(packageName) to see what package manager was used to install the application.
  • The user installed an APK that uses libraries that were not compiled for their target architecture. Using x86_64 libs on an armeabi-v7a device for example. You can use Build.SUPPORTED_ABIS to query the userā€™s ABI and check it against the APKā€™s target architecture.

If you could incorporate these queries into your telemetry that might provide more insights into the crash. Have you been able to reproduce this crash locally?

43

@jeff_fmod thanks for the response. We have not been able to reproduce the crash locally. That being said, about 1% of users are getting this crash in production. I guess it could be due to users side loading the app from elsewhere then Google Play.

I will add logs for the install source, as well as the comparisons of ABIs. I will also try preloading the libs as you suggested and report back!

44

Hello, we are experiencing this issue again.
2.02.27 C++ API, OpenSL, Android 9-15. Android 15 devices for example: vivo Y200e, Redmi Note 12 5G, Samsung Galaxy A04e.
You mentioned earlier that it should be fixed in Android 13, but I as we see itā€™s not.
Here is the stack trace:

* libc +0x088b78 __memcpy
* split_config.arm64_v8a.apk +0x13947c FMOD::Geometry::getUserData
* split_config.arm64_v8a.apk +0x0d3230 FMOD::SystemI::createDiskFile
* split_config.arm64_v8a.apk +0x150a2c Java_org_fmod_FMOD_OutputAAudioHeadphonesChanged
* libwilhelm +0x01c188 <unknown>
* libaudioclient +0x08af00 android::AudioTrack::processAudioBuffer
* libaudioclient +0x08a0f0 android::AudioTrack::AudioTrackThread::threadLoop
* libutils +0x0154d4 android::Thread::_threadLoop
* libandroid_runtime +0x0a4974 android::AndroidRuntime::javaThreadShell
* libutils +0x014db0 <unknown>
* libc +0x0f40c4 __pthread_start
* libc +0x08ed10 start_thread

Do you have any ideas about it?
Thanks in advance.

45

This looks like a different crash to me. Although there are no symbols to compare against, the libs should be accurate- the Android 13 crash occurs entirely on the OS side, whereas this crash is from OpenSL (libwilhelm) calling into your apk (split_config) and failing during a memcpy.

Unless you have some other lib in your apk that plays audio (such as a video player), the crash is most likely coming from FMODā€™s OpenSL feeder thread, which would be unrelated to that Android 13 crash.
As for why OpenSL is crashing while the feeder thread is running, I think Iā€™m going to need more information. Can you please tell me:

  1. Are you only getting this exact crash after updating to 2.02.27?
  2. What system are you using for your crash reporting?
  3. Are you seeing any crashes since updating that have libaaudio_internal in the call stack?
  4. How frequent is this crash, and can you reproduce it locally?
46

Thanks for reply.

  1. We are on 2.02.27 from March, but I have history only from June, so itā€™s difficult to say now. All this time it exists.
  2. Sentry.
  3. I can not find such.
  4. We can not reproduce it. But our users have it quite frequently, but not very.
47

Thanks for the extra info. Since you arenā€™t seeing anything in libaaudio_internal, this is definitely a different crash.
We do have one outstanding Android issue:

It is important that you do not call any FMOD API functions after System::mixerSuspend other than System::mixerResume, even if you intend to shutdown FMOD as you may cause a deadlock.

All of the Android crashes I have investigated recently have been related to this issue, and if you update to the latest version of FMOD there are a few specific cases that are now handled. However, there are still many cases that arenā€™t. I think the best course of action for you would be:

  1. Update FMOD to the latest version (2.02.30) to avoid some of the common mixerSuspend cases weā€™ve identified
  2. Check your code for any calls to System::mixerSuspend and make sure there are no other threads that can possibly call into the FMOD API after System::mixerSuspend has been called
  3. Upload FMOD debug symbols to Sentry so that you can get usable call stacks after a crash. I have uploaded symbols for both 2.02.27 and 2.02.30 to your FMOD Profile. I am unsure what the process is for Sentry, but I believe it would just be a matter of following the steps here Uploading Files | Sentry for Android.
1 Like
48

We have applied your debug symbols and now the stack trace looks like this:

split_config.arm64_v8a.apk +0x143640 FMOD_File_GetDiskBusy
* split_config.arm64_v8a.apk +0x0e7a8c FMOD::SystemI::createDiskFile
* split_config.arm64_v8a.apk +0x0e7ae4 FMOD::SystemI::createDiskFile
* split_config.arm64_v8a.apk +0x0e7c98 FMOD::SystemI::createDiskFile
* split_config.arm64_v8a.apk +0x133738 FMOD::ChannelGroup::getChannel
* split_config.arm64_v8a.apk +0x104e44 FMOD::SystemI::createDiskFile
* split_config.arm64_v8a.apk +0x11d0f4 FMOD::DSPConnection::getUserData
* split_config.arm64_v8a.apk +0x11cd40 FMOD::DSPConnection::getUserData
* split_config.arm64_v8a.apk +0x11c874 FMOD::DSPConnection::getUserData
* split_config.arm64_v8a.apk +0x1281cc FMOD::DSPConnection::getUserData
* split_config.arm64_v8a.apk +0x127720 FMOD::DSPConnection::getUserData
* split_config.arm64_v8a.apk +0x1381fc FMOD::Geometry::getUserData
* split_config.arm64_v8a.apk +0x137540 FMOD::Geometry::getUserData
* split_config.arm64_v8a.apk +0x1373b8 FMOD::Geometry::getUserData
* split_config.arm64_v8a.apk +0x11d0f4 FMOD::DSPConnection::getUserData
* split_config.arm64_v8a.apk +0x11cd40 FMOD::DSPConnection::getUserData
* split_config.arm64_v8a.apk +0x137fe4 FMOD::Geometry::getUserData * split_config.arm64_v8a.apk +0x1509c4
Java_org_fmod_FMOD_OutputAAudioHeadphonesChanged
* split_config.arm64_v8a.apk +0x147e00 FMOD_Thread_SetAttributes
* split_config.arm64_v8a.apk +0x14c3e4 FMOD_OS_Time_Sleep
* libc +0x0e205c __pthread_start
* libc +0x084af0 __start_thread

I hope it will help.

49

Thank you for the call stack, were you able to test updating or checking for any calls to the FMOD API after System::mixerSuspend was called?

50

We donā€™t have any calls to FMOD API after calling System::mixerSuspend for sure.
We havenā€™t checked the newest API yet. I think it wonā€™t be done for the next several months.

51

Thank you for confirming. Unfortunately, I have not been able to reproduce an issue with a similar callstack. Is there any more information about the current state of the app when the crash happens, e.g. is it in the background?

Would it be possible to enable fmod logging: FMOD Engine | Core Api Common - Debug::Initialize and share any FMOD logs?

52

Hello, now we have some FMOD debug logs before crash. Iā€™ve attached some.
Also the stacktrace has been changed, now itā€™s like this:

0   libc.so                         0x7f7b0f9ce0        <unknown> + 547525467360
1   split_config.arm64_v8a.apk      0x7d9b15947c        FMOD::OutputRingBuffer::read
2   split_config.arm64_v8a.apk      0x7d9b0f3230        FMOD::OutputRingBuffer::readSamples (fmod_outputi.h:88)
3   split_config.arm64_v8a.apk      0x7d9b170a2c        FMOD::OutputOpenSL::feederThread (fmod_output_opensl.cpp:479)
4   libwilhelm.so                   0x7f7288d5ec        <unknown> + 547382416876
5   libaudioclient.so               0x7f8a9b2328        android::AudioTrack::processAudioBuffer
6   libaudioclient.so               0x7f8a9b14fc        android::AudioTrack::AudioTrackThread::threadLoop
7   libutils.so                     0x7f5eaacca4        android::Thread::_threadLoop
8   libandroid_runtime.so           0x7f94f8fb0c        android::AndroidRuntime::javaThreadShell
9   libc.so                         0x7f7b111ca8        <unknown> + 547525565608
10  libc.so                         0x7f7b1033c0        <unknown> + 547525505984

Or this:

0   libc.so                         0xf51789dc          __memcpy_a53
1   split_config.armeabi_v7a.apk    0xc26b2e20          FMOD::OutputRingBuffer::read
2   split_config.armeabi_v7a.apk    0xc265a11a          FMOD::OutputRingBuffer::readSamples (fmod_outputi.h:88)
3   split_config.armeabi_v7a.apk    0xc26c5a0e          FMOD::OutputOpenSL::feederThread (fmod_output_opensl.cpp:479)
4   libwilhelm.so                   0xf50151e9          <unknown> + 4110504425

log3.txt (14.3 KB)
log1.txt (599 Bytes)
log2.txt (3.4 KB)

1 Like
53

Thank you for sending those over. It looks like the output is still running after mixerSuspend has been called, which is most likely due to a known issue in 2.02.27 which was partially fixed in 2.02.28, and should be fully fixed in 2.02.29:

2.02.28: Core API - Android - Fixed crash when calling into record API while mixer suspended.
2.02.29: Core API - Fixed deadlock when adding/removing audio device after call to System::mixerSuspend.

I am confident that if you update it will resolve these crashes. Can you please try updating to the latest version of the FMOD API?

1 Like