Asset files deleted when using "Repair Asset Conflicts" button

richardtaylor · July 5, 2021, 6:11am

When opening FMOD projects sent to us from our audio designer I often see the following message:

There are assets in the project with conflicting compression and streaming settings. You can fix the issue by using the “Repair Path Conflict…”

I then go to the Assets tab, right-click and select “Repair Asset Conflicts”. Whenever I do this, it seems that FMOD Studio’s idea of “repairing” is to just delete the offending file. It remains listed in the Assets view, but it can’t be previewed and when I look for it in Explorer it’s just gone. If I use Explorer to copy the file from elsewhere to replace it everything is fine again.

The question is, why is this happening at all?

Our audio designer can do a build with no errors reported. They’re working on OS X.
They then make an .fspackage and send it to me.
I open it on my Windows workstation and press build, without having changed anything. I get the error somewhat regularly in this manner.
And while I assume it isn’t deliberate, FMOD Studio “fixes” it by… deleting the file?

joseph · July 6, 2021, 5:40am

Can you confirm that the missing asset files are present before you attempt to repair asset conflicts?

richardtaylor · July 6, 2021, 7:25am

Hi joseph. Yes, as pictured, the file is definitely present when the conflict is flagged:

And here is the same folder a few seconds after I pressed “Repair” and the dialog closed:

This is in FMOD Studio 2.01.10.

joseph · July 8, 2021, 4:37am

After some investigation, I’ve discovered that this issue occurs when two different asset metadata files point to the same audio file. In theory, this should never happen.

How are you getting the copy of your project from your audio designer? Are you using some form of source control, and if so, what? Or are you send a copy of the project as a compressed archive that you then extract, and if so, do you copy it into the same directory as an existing version of the project in order to overwrite it?

richardtaylor · July 8, 2021, 8:17am

We’re using File → Package Project, and then opening that packaged project at the other end.

I would love to use version control, but it’s not something our audio designer is familiar with.

joseph · July 12, 2021, 12:37am

When the package is opened at the other end, it is placed into an empty folder, or is it pasted into a folder that already contains an older version of the project that the new version overwrites?

richardtaylor · July 12, 2021, 4:28am

Hi joseph. It’s an empty folder. Having said that, I’ve just tried it twice now again and not got that error.

I do get a pop-up asking if I want to recover the project because a bunch of stuff is missing. It claims to do this successfully, and then carries on, including being able to build without raising any errors.

Digging further, the “Repair Asset Conflicts” thing happens afterwards, when I do put it into our game project by copying it over an earlier version. So the issue here is that the asset has been (re)imported separately between the project getting sent off and coming back?

joseph · July 13, 2021, 1:50am

Yes and no. The asset metadata issue you’ve encountered is likely just one of many project invalidities created when you overwrite an old version of your FMOD Studio project with a new one.

An FMOD Studio project consists of a large number of files, many of which reference each other in complex and interconnected ways. (For example, the metadata file for an event needs to reference the metadata files for the folder containing that event, the assets used by that event, and the bus into which the event is routed; it may also reference the metadata for specific preset effects and parameters used in the event and the buses targeted by its sends and sidechain effects.) Pasting a new version of an FMOD Studio project into the same folder as an existing older version overwrites any files in the old version that are also present in the new version, but not files in the old version that aren’t present in the new version - so you’ll end up with a Frankenstein’s monster of new files and old ones, many of which contain outdated references to things that have since been changed or no longer exist.

We call these contradictory and broken references “project invalidities,” and when FMOD Studio detects them, it prompts you to repair them. Unfortunately, invalid metadata is by definition contradictory and incomplete, making it impossible for FMOD Studio to know what the invalid content is supposed to be like - so it has to make some assumptions, and those assumptions will not always match your original intent. Thus, while a repaired project is always free of invalid content, it may also be changed in ways you didn’t want.

In other words, Frankensteining the old and new versions of your project into one is a bad idea. Fortunately, you can easily avoid doing this by instead deleting the old version of the project, then pasting the new version into the emptied folder.

richardtaylor · July 13, 2021, 2:19am

Fair enough. As we’re not removing anything between versions I didn’t anticipate that there would be any cases of “files in the old version that aren’t present in the new version”.

Anyhow, if always deleting the contents of the project folder before dropping in an update is the go then that’s easy enough.

On a related note, do you have any ideas of why we might need to do a project recovery after packaging to .fsproject on OS X then unpacking it into a fresh folder on Windows?

Thanks for your assistance.

joseph · July 13, 2021, 5:09am

It’s not always necessary to remove content for this kind of issue to occur. To give a couple of examples:

Renaming an asset can result in newer versions of a project including a file that doesn’t have the same name as the file did in older versions of the project, and thus in that older version of the file not being overwritten.
Importing an asset generates a new metadata file for that asset, and metadata file names are GUIDs. This means that if an asset is deleted and then re-imported, the new metadata file will have a different GUID, and so will not overwrite older versions of the asset’s metadata file from before it was deleted and re-imported, even if the asset file itself is the same.

A packaged project is just a compressed archive containing a project’s essential files. “Essential” is the key word there: When creating a package, FMOD Studio deliberately excludes non-essential data that do not affect in-game behavior. (The excluded data is mostly machine-specific user settings that it would not make sense to duplicate on other machines.) When a packaged project is loaded, FMOD Studio detects the missing data and prompts you to “Recover” it - which usually means regenerating it from scratch.

It works this way because the “package project” feature was meant to allow users to create stripped-down versions of their projects to send to us, so that we can more easily investigate issues and reproduce reported bugs. Users opening projects they’d sent each other was something we didn’t expect, which is why opening a packaged project has this user-unfriendly quirk.

That being said, there should be no problems caused by transferring packages between users as you have been doing, as long as your audio designer doesn’t exclude a anything from the package that you actually need.

rhys_vdw · July 15, 2021, 2:01am

Hi, I’d just like to report that I’m running into this issue right now. I suspect it’s happened as a result of merging different audio feature branches in git (without any manual conflict resolution). But I cannot be sure.

FMOD Studio 2.01.01 Build #109257

Happy to share FMOD project or relevant commit history privately if helpful.

Detailed description

Same error presented on build:

(Also can I suggest a word wrap on this dialogue?)

There are assets in the project with conflicting compression and streaming settings. You can fix the issue by using the “Repair Path Conflict…” context menu option in the Assets Browser.

Here is the “repair asset conflict” dialogue:

Result of hitting “Repair”

$ git status
On branch 1691-speed-up-burble
Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)        
  (use "git restore <file>..." to discard changes in working directory)
        deleted:    FmodProject/Assets/StupidCustomer_S2_1.wav
        deleted:    FmodProject/Assets/StupidCustomer_S2_2.wav
        deleted:    FmodProject/Assets/StupidCustomer_S2_3.wav
        deleted:    FmodProject/Assets/StupidCustomer_S2_4.wav
        deleted:    FmodProject/Assets/StupidCustomer_S3_1.wav
        deleted:    FmodProject/Assets/StupidCustomer_S3_2.wav
        deleted:    FmodProject/Assets/StupidCustomer_S3_3.wav
        deleted:    FmodProject/Assets/StupidCustomer_S3_4.wav
        deleted:    FmodProject/Metadata/AudioFile/{1ce8dc6e-6413-42fa-9b8d-58b1eb06621e}.xml
        deleted:    FmodProject/Metadata/AudioFile/{2e89b2b4-2588-4e17-b727-8591a757ad7b}.xml
        deleted:    FmodProject/Metadata/AudioFile/{3e4524a7-9c9e-46b1-93bb-4fe998c26fca}.xml
        deleted:    FmodProject/Metadata/AudioFile/{4a9144c2-1d98-4a3c-8cd8-c8ae82a68de1}.xml
        deleted:    FmodProject/Metadata/AudioFile/{55106930-10d2-47a2-905f-dd0845a2720f}.xml
        deleted:    FmodProject/Metadata/AudioFile/{55ae3942-88fe-4db0-b892-4147be91ea73}.xml
        deleted:    FmodProject/Metadata/AudioFile/{701a9d35-1616-46f6-87da-038b435fb933}.xml
        deleted:    FmodProject/Metadata/AudioFile/{97a2361b-3b37-4dd8-8935-dab87e4067c6}.xml

One example deleted XML files:

Metadata\AudioFile\{1ce8dc6e-6413-42fa-9b8d-58b1eb06621e}.xml:

<?xml version="1.0" encoding="UTF-8"?>
<objects serializationModel="Studio.02.01.00">
	<object class="AudioFile" id="{1ce8dc6e-6413-42fa-9b8d-58b1eb06621e}">
		<property name="assetPath">
			<value>StupidCustomer_S3_4.wav</value>
		</property>
		<property name="frequencyInKHz">
			<value>48</value>
		</property>
		<property name="channelCount">
			<value>2</value>
		</property>
		<property name="length">
			<value>0.27035416666666667</value>
		</property>
		<relationship name="masterAssetFolder">
			<destination>{44cafa7b-9607-40b2-9272-96b72582dc1e}</destination>
		</relationship>
	</object>
</objects>

Affected events still seem to know the name of the assets despite them having been deleted:

Outcome

I can’t find any errors as a consequence of ignoring this warning, build succeeds and audio sounds fine in Unity editor. At the moment I’d just like to be able to clear the error and understand if there are any risks. I’d be happy to manually modify the XML files to fix the problem too.

Thanks!

rhys_vdw · July 15, 2021, 2:22am

Ah right, I think the answer is here. And indeed we have ended up with redundant asset metadata files for each of the assets that are deleted.

I can probably manually repair this problem then.

Perhaps stating the obvious, but I think the “repair” operation should detect this case, dedupe the asset metadata files and update project references instead of just deleting the used assets outright. “repair asset paths” doesn’t sound like it’s going to actually delete your audio files.

Anyway, hope my report was helpful.

richardtaylor · July 15, 2021, 3:01am

Indeed, under no circumstances should asset files be deleted without user confirmation. I understand that one of the meta files has to go, but the assets themselves should not be deleted.

Ideally, we’d also be asked about which metadata file we want to keep, unless they’re identical. And if they’re identical then they’re not “conflicting”, they’re “redundant”.

joseph · July 15, 2021, 4:18am

You are, of course, correct.

I’ve already added a task to our tracker to revisit the reasoning behind this behavior and identify ways of making it safer for users.

richardtaylor · July 15, 2021, 4:45am

Great to hear.

rhys_vdw · July 19, 2021, 2:16am

Hey Joseph, an update here.

So I tried to resolve this issue. What I did was one-by-one go through each meta file that contained duplicate filenames (the ones that were being deleted by the repair tool). For each one I would remove one of the files and then check whether the file had been stripped from its event. When I found the one that was not being used by the event I’d commit that.

So ultimately i ended up without any duplicate, and AFAICT the meta files that were actually referenced by the project being used.

However after doing this the error still persisted. So I deleted all metafiles for these files (including that were in use by the event). Then I let FMOD re-detect these audio files by restarting and reassigned them into the events that had been stripped.

This still has not cleared the warning. So I think the problem might be a bit more complicated than just the redundant meta files.

Project is still building fine, but I’m wondering if you have any suggestions for what we might try? The warning suggests something to do with “compression settings”, can I change these somewhere?

rhys_vdw · July 22, 2021, 3:56am

Hm, scratch that. I’m not sure what I screwed up above but I was able to fix the problem using the following steps:

Run repair paths tool
Revert to saved
Revert deletion of asset files (wavs, not the xml meta files)
In assets view select “missing” files and “refresh assets”

After that the build errors are cleared. The result is one set of duplicate meta files being deleted. Still not sure how the error came about in the first place but it seems I’ve resolved the issue on our end.

It does seem as though the problem here is just that the assets are being deleted after the tool is run.

Thanks!

richardtaylor · August 7, 2021, 6:31am

Somewhat related to conversation in here already, is there any valid way using the “Package Project” function that my sound designer could have sent me a .fspackage folder which does not have a .fspro file in it? Because that’s exactly what I’m looking at now.

joseph · September 2, 2021, 11:55pm

No, that should be impossible. Even if you uncheck everything when creating a package, the .fspro will still be in it.

Could your sound designer have manually created an archive from some of the project’s files, and changed its filename extension?

richardtaylor · September 13, 2021, 6:13am

No, I really can’t imagine them doing that.