I ended up simply flipping the up-vector when calling setListenerAttributes() while in mirrored mode.