Sync actions to audio in a rhythm game

[Sapryx]

I’m working on a rhythm game where the player needs to press certain buttons to the beat to perform different actions. Depending on how accurately to the beat you do it, the result of those actions changes (for example, the damage of an attack).
I’ve set up a system of callbacks to track the current playback position and call an even when the callback happens. But then something strange happens: in the function subscribed to this event I get the current playback position using Studio::EventInstance::getTimelinePosition, but it doesn’t match the position of the beat somehow, it’s always slightly offset. And what’s even stranger is that the offset is always different.
I think I’m missing something about how audio engines work under the hood. What should I look for?

[li_fmod]

Hi,

Thank you for sharing the information.

@Sapryx

but it doesn’t match the position of the beat somehow, it’s always slightly offset. And what’s even stranger is that the offset is always different.

Could I please grab more info? What version of FMOD Unity integration are you using? How large is the offset that you are observing between the beat and the user input timeline position?

Due to the asynchronous nature of Studio, there’s usually a very slight difference in timing between most commands being called via the API, and being executed by Studio.

Studio also has a default update period of 20ms, which means that the timeline position will update every 20ms instead of constantly. Both of these can contribute to the beat position being slightly offset from the time retrieved from getTimelinePosition.

For more information, you could also have a look at a similar post here:

Rythm game and latency

Hello everyone! Part of an upcoming project will include a rythm mini-game. My approach so far was to simply play a music event in FMOD, and in parallel inside of Unity trigger pre-determined rythms. It is mostly okay, but as you might expect there is a bit of latency between the two systems that are completely independent, and it gets worse with frame drops. I did see we could edit the DSP Buffer Size to diminsh latency, but the documentation seems to say it shouldn’t be used, and since this m…

Hope this helps, let me know if you have any questions.

[Sapryx]

Thank you for your response. I am using version 2.02.22 version of FMOD Unity integration.
The offset varies around 6-60 ms for a 120 bpm song. The way I do the calculations is the following:

1. I have a BeatTracker script that updates the current position of the song every frame and calls an event when nessesary.

private void Update() {
	playbackEvent.getPlaybackState(out playbackState);
	
	var isPlaying = playbackState == PLAYBACK_STATE.PLAYING;
	
	if(isPlaying) {
		playbackEvent.getTimelinePosition(out timelineInfo.playbackPosition);

		UpdateDspClock();
		ProcessBeatEvents();
	}
}

private void UpdateDspClock() {
	masterChannelGroup.getDSPClock(out currentSamples, out parentDsp);
	currentDspTime = (double)currentSamples / masterSampleRate;
}

private void ProcessBeatEvents() {
	var beatLength = timelineInfo.BeatLength;
	var dspSongPosition = (float)(currentDspTime - trackDspStartTime);
	var upBeatSongPosition = dspSongPosition + beatLength / 2f;
	
	if(dspSongPosition >= lastDownbeatTime + beatLength) {
		var correctionAmount = Mathf.Repeat(dspSongPosition, beatLength);
		
		lastDownbeatTime = dspSongPosition - correctionAmount;
		lastDownbeatDspTime = currentDspTime - correctionAmount;
		
		OnDownbeat?.Invoke();
	}

	...
}

2. In Shooting script I have a function to start the song and invoke the repeating function.

private void StartMusic() {
	RhythmHandler.StartMusic();
	InvokeRepeating(nameof(ShootTest), 0f, 1f);
}

private void ShootTest() {
	RhythmHandler.IsOnBeat(beatAccuracy);
}

3. IsOnBeat calculates different values and tells how accurately to the beat the action was. This is the important part. Here I compare the timelinePosition (currentTime) to the time of the nearest downbeat (currentBeatTime). The function is being called every second and started being called with the song. The song is in 120 bmp, so downbeats are exactly 1 second apart and I expect the currentTIme and the currentBeatTime to match and nearestBeatDistance to be 0, which is not the case. In result, the values are alway offset and it’s not consistent.

public static BeatHitResult IsOnBeat(float accuracy) {
	var timelineInfo = BeatTracker.instance.timelineInfo;
	var currentBeatTime = BeatTracker.instance.lastDownbeatTime;
	var nextBeatTime = currentBeatTime + timelineInfo.BeatLength;
	var currentTime = timelineInfo.playbackPosition / 1000f;

	var leftBeatDistance = Math.Abs(currentTime - currentBeatTime);
	var rightBeatDistance = Math.Abs(nextBeatTime - currentTime);
	var nearestBeatDistance = Math.Min(leftBeatDistance, rightBeatDistance);
	
	print($"currentTime: {currentTime}, currentBeatTime: {currentBeatTime}");
	print($"left: {leftBeatDistance}, right: {rightBeatDistance}, nearest: {nearestBeatDistance}");

	...
}

Am I doing it wrong and what is the best way to achieve what I aim for (if it’s even possible)? Maybe track time in unity separately?

[li_fmod]

Thank you for sharing the code and additional information.

@Sapryx

I have a BeatTracker script that updates the current position of the song every frame and calls an event when nessesary.

The use of Unity’s Update() function is frame-rate dependent, which means it might be introducing a tiny delay and the exact timing can vary between frames.

Instead of relying on Unity’s Update() function to check the beat every frame, you could consider using Timeline callbacks to sync the beats more precisely. These callbacks are tied directly to the FMOD DSP clock, so you can use them to handle downbeat events with more accuracy.