To follow with an even more clear exemple of this, here’s a basic implementation, done in a UE4 template. It highlights the same issue as above, but in a different manner. To be listened with headphones.
A speaker, placed bellow the player, and offset on the left-right axis, diffuses some music. As the player rotates around the left-right axis, you can hear the panning is changing, though it shouldn’t. Most of the time, the panning is normal, mostly centered, but when facing forward, the sound is abruptly hard panned right.