FMOD 2.0.3.7 + Unity 2022.3.62f2 + GitLab Runner fails to build

FMOD Engine
1

Hello,

we are using a GitLab Runner instance to build our project for Windows, Mac and Linux.

After a recent update to the patched Unity Version, our projects are not building anymore.

GitLab Runner is using shell scripts / Makefiles to build, and it tries to rebuild the EventCache:

FMOD: Event cache is missing or in an old format; creating a new instance.

This fails as such:

Asset Pipeline Refresh (id=bc432898d6a1f4d1288d6fd928ba4020): Total: 0.055 seconds - Initiated by StopAssetImportingV2(NoUpdateAssetOptions)
Build Task PostDependencyCallback failed with exception:
Value cannot be null.
Parameter name: key
at System.Collections.Generic.Dictionary2[TKey,TValue].FindEntry (TKey key) [0x00008] in <9c9d7808da6a43e3b68409f2b63b3e6f>:0 at System.Collections.Generic.Dictionary2[TKey,TValue].ContainsKey (TKey key) [0x00000] in <9c9d7808da6a43e3b68409f2b63b3e6f>:0
at FMODUnity.EventCache+<>c__DisplayClass16_0.b__0 (FMODUnity.EditorEventRef eventRef) [0x00000] in /home/gitlab-runner/builds/TDz60_OfF/0/shinydynamics/tidalrush/TidalRush/Assets/Plugins/FMOD/src/Editor/EventCache.cs:80
at System.Collections.Generic.List1[T].ForEach (System.Action1[T] action) [0x0001e] in <9c9d7808da6a43e3b68409f2b63b3e6f>:0
at FMODUnity.EventCache.BuildDictionary () [0x0001f] in /home/gitlab-runner/builds/TDz60_OfF/0/shinydynamics/tidalrush/TidalRush/Assets/Plugins/FMOD/src/Editor/EventCache.cs:79
at FMODUnity.EventManager.UpdateCache () [0x0091e] in /home/gitlab-runner/builds/TDz60_OfF/0/shinydynamics/tidalrush/TidalRush/Assets/Plugins/FMOD/src/Editor/EventManager.cs:358
at FMODUnity.EventManager.RefreshBanks () [0x00000] in /home/gitlab-runner/builds/TDz60_OfF/0/shinydynamics/tidalrush/TidalRush/Assets/Plugins/FMOD/src/Editor/EventManager.cs:32
at FMODUnity.EventManager.BuildTargetChanged () [0x00000] in /home/gitlab-runner/builds/TDz60_OfF/0/shinydynamics/tidalrush/TidalRush/Assets/Plugins/FMOD/src/Editor/EventManager.cs:1055
at FMODUnity.EventManager+<>c.<.cctor>b__15_0 () [0x00000] in /home/gitlab-runner/builds/TDz60_OfF/0/shinydynamics/tidalrush/TidalRush/Assets/Plugins/FMOD/src/Editor/EventManager.cs:572
at FMODUnity.BuildStatusWatcher.SetBuildInProgress (System.Boolean inProgress) [0x0003c] in /home/gitlab-runner/builds/TDz60_OfF/0/shinydynamics/tidalrush/TidalRush/Assets/Plugins/FMOD/src/Editor/EditorUtils.cs:1747
at FMODUnity.BuildStatusWatcher+<>c.b__6_0 (UnityEditor.Build.Pipeline.Interfaces.IBuildParameters parameters, UnityEditor.Build.Pipeline.Interfaces.IDependencyData dependencyData) [0x00000] in /home/gitlab-runner/builds/TDz60_OfF/0/shinydynamics/tidalrush/TidalRush/Assets/Plugins/FMOD/src/Editor/EditorUtils.cs:1788
at UnityEditor.Build.Pipeline.BuildCallbacks.PostDependency (UnityEditor.Build.Pipeline.Interfaces.IBuildParameters buildParameters, UnityEditor.Build.Pipeline.Interfaces.IDependencyData dependencyData) [0x00008] in ./Library/PackageCache/com.unity.scriptablebuildpipeline@1.21.25/Editor/Shared/BuildCallbacks.cs:45
at UnityEditor.Build.Pipeline.Tasks.PostDependencyCallback.Run () [0x00000] in ./Library/PackageCache/com.unity.scriptablebuildpipeline@1.21.25/Editor/Tasks/PostDependencyCallback.cs:28

Builds work locally from the Editor, as well as from an actual Build machine when using the Editor instead of headless.

I’ve tried multiple fixes, but not getting any further with this, so any help may be appreciated.

2

Some more experiments:

  • Updating to 2.0.3.9 didn’t help

  • Stopping the gitlab-runner mid task, and then connecting to the runner, changing to the runner user, sourcing the envirment and starting the exact same make script: Build works.

I’m a bit stumped at the last part, because that’s more or less the same procedure. One thing I could think of is that maybe the runners don’t change into the correct directory and when trying to refresh things it doesn’t find the files? But that’d have to be relative path configurations that I don’t really see at the moment.

I could also “force” the error in the editor by deleting the Library folder, but a subsequent build or run from the editor doesn’t have any issues.

3

More update:

  • Changing to Unity 6000.2.6f2 didn’t fix the issue.
  • Gitlab runner versions have not changed.
  • Reproduces on Linux, Mac and WIndows Gitlab- Runners.
  • Manually triggering the build works on the same repos that failed while running in a runner.
4

Hi,

We’ve seen a few other errors related to CI builds in the past, but those should be fixed as of 2.02.30/2.03.09. Just to confirm a few things:

  • Are the banks belonging to your FMOD project present in your GitLab Runner instance?
  • Does the system you’re running the instance on have audio capabilities? i.e. functioning audio drivers and output devices
  • What is the full build command the runner is executing?
  • Could I get you to provide a full log from your instance, not just the exception you’ve already posted?
5

Hi again,

sure, I’ll gladly get you up to date with your questions and some more research results:

  • Yes, the banks are in the repo and available. (See below)
  • One does not, the other two do have audio capabilities.We use a Linux, a Mac, and a Windows machine. Linux is server (no audio), the others are normal desktop machines with audio installed. Issue is the same across all platforms, and workaround also works on all of them.
  • “$UNITY” -projectPath ./$PROJECTNAME -quit -batchmode -executeMethod Base.BuildScript.Build -buildPath Build/$2 -buildFileName $3 -logFile ${PROJECTNAME}/Build/$2/Editor.log
  • I’ll check if I can attach a full log, I will get back to you on that.

More research was done:

Initially, we thought that running the command manually would fix the issue, but on further research that was incorrect.

Starting the build twice will result in a complete build. So as a workaround we just start the build twice, back to back.

So it seems that somehow, FMOD throws an error on first build/import (Due to no Temp/ or Library Folder for example) which causes the build to fail where it could technically continue, so issuing a second run (with Temp/ Library/ at the point of exit) will pick up where the first build stopped and then finish the build. The Editor probably does that first refresh/import on itself and then the Build succeeds? Just guessing.

Even though we have a workaround, we’ll continue investigating.

6

Sorry for the delayed response, and thanks for providing so much info on the issue.

Unfortunately, I haven’t been able to reproduce the issue on my end - I’m able to use both a manual shell command, and a GitLab runner instance, to build a Unity project using the command you shared without any issues, even with the FMOD cache and any temporary Unity files/directories deleted.

The full build log would be great - I also have few more questions:

  • What version of GitLab runner are you using?
  • Has FMOD for Unity been installed using the Unity asset store, or using a direct download from our website?
  • Are you able to reproduce the event cache issue, or any other issues, when building from the Unity editor?
  • Can I get you to share any code executed by Base.BuildScript.Build?

If confidentiality is an issue, you can upload any files to the uploads tab of your FMOD user profile, which is only accessible by yourself and FMOD staff.