How to sync notes to music in Unity (and fix latency)

Drift: notes slowly slide out of time

If notes start in time and get worse over the song, you are accumulating frame time. Compute the song position from the audio clock every frame instead:

songPosition = (float)(AudioSettings.dspTime - dspStart - pausedTime) * music.pitch;

Track the total time spent paused (pausedTime) and subtract it, or every pause will shift the whole chart.

Constant offset: everything is slightly late

If notes are consistently early or late by the same amount, the cause is output latency: the time between Unity starting a sound and it leaving the speakers. Bluetooth headphones can add 150 ms or more. Two fixes:

• Make sure the chart's zero point matches the audio. An MP3 encoder can add silence at the start, so check the first note against the waveform.
• Give players a calibration offset: play a metronome, have them tap along, and use the average difference as their offset.

Jitter: notes stutter even though timing is right

AudioSettings.dspTime updates in audio-buffer steps, so positions computed from it can move unevenly between frames. Smooth the visual position (for example, lerp towards the target) while keeping hit judgement on the true song time. Smoothing the visuals is fine, but never smooth the time you judge hits against.

Start playback cleanly

Start the music with AudioSource.PlayScheduled a short moment in the future rather than Play(), and show a countdown during that lead-in. A scheduled start lands on an exact DSP time, so frame timing can't shift the start.

Where these live in Rhythm Game Starter

• SongManager computes song position from AudioSettings.dspTime, minus a start delay and accumulated pause time, scaled by pitch for speed modifiers. It can also pause and resume automatically when Time.timeScale changes.
• TrackManager has a hitOffset for latency tuning, two sync modes (move the whole track, or each note) and optional sync smoothing.
• Full source, so you can add a player-facing calibration screen on top. See what's included.